Program A: malloc memory but do reference any data inside the allocated memory (mallocaed but not used):
#include <stdio.h>
#include <stdlib.h>
#define MEGABYTE 1024*1024
int main(int argc, char *argv[])
{
void *myblock = NULL;
int count = 0;
while (1)
{
myblock = (void *) malloc(MEGABYTE);
if (!myblock) break;
printf("Currently allocating %d MB\n", ++count);
}
exit(0);
}
Compile the program, run it, and wait for a moment. Sooner or later it will go OOM. Now compile the program B, which allocates huge blocks and fills them with 1:
#include <stdio.h>
#include <stdlib.h>
#define MEGABYTE 1024*1024
int main(int argc, char *argv[])
{
void *myblock = NULL;
int count = 0;
while(1)
{
myblock = (void *) malloc(MEGABYTE);
if (!myblock) break;
memset(myblock,1, MEGABYTE);
printf("Currently allocating %d MB\n",++count);
}
exit(0);
}
Notice the difference? Likely, program A allocates more memory blocks than program B does. It's also obvious that you will see the word "Killed" not too long after executing program B. Both programs end for the same reason: there is no more space available. More specifically, program A ends gracefully because of a failed
malloc(). Program B ends because of the Linux kernel's so-called OOM killer.
The first fact to observe is the amount of allocated blocks. Assume that you have 256MB of RAM and 888MB of swap (my current Linux settings). Program B ended at:
Currently allocating 1081 MB
On the other hand, program A ended at:
Currently allocating 3056 MB
Where did A get that extra 1975MB? Did I cheat? Of course not! If you look closer on both listings, you will find out that program B fills the allocated memory space with 1s, while A merely simply allocates without doing anything. This happens because Linux employs deferred page allocation. In other words, allocation doesn't actually happen until the last moment you really use it; for example, by writing data to the block. So, unless you touch the block, you can keep asking for more. The technical term for this is optimistic memory allocation.
Here is session when each program was run:
|14:59:41|ec2-user@mybox:[tmp]> cat memA.c #include <stdio.h> #include <stdlib.h> #define MEGABYTE 1024*1024 int main(int argc, char *argv[]) { void *myblock = NULL; int count = 0; while (1) { myblock = (void *) malloc(MEGABYTE); if (!myblock) break; printf("Currently allocating %d MB\n", ++count); } exit(0); } |14:59:49|ec2-user@mybox:[tmp]> free -tam total used free shared buffers cached available Mem: 7483 349 7134 0 10 63 7098 -/+ buffers/cache: 275 7208 Swap: 8191 741 7450 Total: 15675 1090 14585 |14:59:56|ec2-user@mybox:[tmp]> grep swap /etc/fstab /swapfile swap swap defaults 0 0 |15:00:02|ec2-user@mybox:[tmp]> ls -l /swapfile -rw-r--r-- 1 root root 8589934592 Nov 17 10:07 /swapfile |15:00:06|ec2-user@mybox:[tmp]> ./memA Currently allocating 1 MB Currently allocating 2 MB Currently allocating 3 MB Currently allocating 4 MB Currently allocating 5 MB Currently allocating 6 MB Currently allocating 7 MB <many lines deleted> Currently allocating 2495385 MB Currently allocating 2495386 MB Currently allocating 2495387 MB Currently allocating 2495388 MB Killed
# Simultaneously on another screen get memory stats |14:54:19|ec2-user@mybox:[~]> while true > do > free -tam > sleep 10 > done total used free shared buffers cached available Mem: 7483 468 7015 0 21 178 7071 -/+ buffers/cache: 268 7215 Swap: 8191 732 7459 Total: 15675 1201 14474 total used free shared buffers cached available Mem: 7483 1582 5900 0 21 178 5956 -/+ buffers/cache: 1382 6101 Swap: 8191 732 7459 Total: 15675 2315 13359 total used free shared buffers cached available Mem: 7483 4042 3441 0 21 178 3497 -/+ buffers/cache: 3842 3641 Swap: 8191 732 7459 Total: 15675 4775 10900 total used free shared buffers cached available Mem: 7483 5800 1683 0 21 178 1739 -/+ buffers/cache: 5599 1883 Swap: 8191 732 7459 Total: 15675 6533 9142 total used free shared buffers cached available Mem: 7483 7369 114 0 0 8 55 -/+ buffers/cache: 7360 123 Swap: 8191 766 7425 Total: 15675 8136 7539 total used free shared buffers cached available Mem: 7483 7371 112 0 0 15 56 -/+ buffers/cache: 7355 128 Swap: 8191 1967 6224 Total: 15675 9339 6336 total used free shared buffers cached available Mem: 7483 7387 96 0 0 16 39 -/+ buffers/cache: 7369 113 Swap: 8191 2969 5222 Total: 15675 10356 5319 total used free shared buffers cached available Mem: 7483 7387 95 0 0 16 37 -/+ buffers/cache: 7369 113 Swap: 8191 3963 4228 Total: 15675 11351 4324 total used free shared buffers cached available Mem: 7483 7383 99 0 2 9 37 -/+ buffers/cache: 7372 111 Swap: 8191 5037 3154 Total: 15675 12421 3254 total used free shared buffers cached available Mem: 7483 7387 95 0 4 12 34 -/+ buffers/cache: 7371 112 Swap: 8191 6197 1994 Total: 15675 13585 2090 total used free shared buffers cached available Mem: 7483 7376 107 0 2 12 45 -/+ buffers/cache: 7361 121 Swap: 8191 7267 924 Total: 15675 14644 1031 total used free shared buffers cached available Mem: 7483 7387 96 0 1 10 32 -/+ buffers/cache: 7375 108 Swap: 8191 8191 0 Total: 15675 15579 96 total used free shared buffers cached available Mem: 7483 424 7059 0 6 34 7006 -/+ buffers/cache: 383 7100 Swap: 8191 743 7448 Total: 15675 1167 14508 #-------------------------------------------------------------------------------------- |15:13:41|ec2-user@mybox:[tmp]> cat memB.c #include <stdio.h> #include <stdlib.h> #define MEGABYTE 1024*1024 int main(int argc, char *argv[]) { void *myblock = NULL; int count = 0; while(1) { myblock = (void *) malloc(MEGABYTE); if (!myblock) break; memset(myblock,1, MEGABYTE); printf("Currently allocating %d MB\n",++count); } exit(0); } Currently allocating 1 MB Currently allocating 2 MB Currently allocating 3 MB Currently allocating 4 MB Currently allocating 5 MB Currently allocating 6 MB Currently allocating 7 MB <many lines deleted> Currently allocating 14380 MB Currently allocating 14381 MB Currently allocating 14382 MB Currently allocating 14383 MB Currently allocating 14384 MB Currently allocating 14385 MB Currently allocating 14386 MB Currently allocating 14387 MB Killed# Simultaneously on another screen get memory stats|15:12:07|ec2-user@mybox:[~]> while true; do free -tam; sleep 10; done total used free shared buffers cached available Mem: 7483 346 7137 0 13 97 7120 -/+ buffers/cache: 234 7248 Swap: 8191 738 7453 Total: 15675 1085 14590 total used free shared buffers cached available Mem: 7483 7385 98 0 0 17 34 -/+ buffers/cache: 7367 116 Swap: 8191 1982 6209 Total: 15675 9367 6308 total used free shared buffers cached available Mem: 7483 7387 96 0 0 17 32 -/+ buffers/cache: 7368 114 Swap: 8191 2879 5312 Total: 15675 10266 5409 total used free shared buffers cached available Mem: 7483 7384 99 0 1 17 35 -/+ buffers/cache: 7366 117 Swap: 8191 3824 4367 Total: 15675 11208 4466 total used free shared buffers cached available Mem: 7483 7385 98 0 1 13 32 -/+ buffers/cache: 7370 113 Swap: 8191 4753 3438 Total: 15675 12138 3537 total used free shared buffers cached available Mem: 7483 7385 97 0 2 13 32 -/+ buffers/cache: 7369 113 Swap: 8191 5762 2429 Total: 15675 13147 2527 total used free shared buffers cached available Mem: 7483 7387 96 0 2 13 31 -/+ buffers/cache: 7371 112 Swap: 8191 6737 1454 Total: 15675 14124 1551 total used free shared buffers cached available Mem: 7483 7384 99 0 3 14 34 -/+ buffers/cache: 7366 117 Swap: 8191 7663 528 Total: 15675 15047 627 total used free shared buffers cached available Mem: 7483 246 7237 0 4 17 7173 -/+ buffers/cache: 225 7258 Swap: 8191 925 7266 Total: 15675 1171 14503 total used free shared buffers cached available Mem: 7483 316 7167 0 6 35 7115 -/+ buffers/cache: 274 7209 Swap: 8191 749 7442 Total: 15675 1065 14610
[root@mybox tmp]# egrep -i 'oom|Out of memory|killed process' /var/log/messages
Apr 18 14:56:52 ip-172-31-10-191 kernel: [13149384.816317] docker invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Apr 18 14:56:52 ip-172-31-10-191 kernel: [13149384.850184] [<ffffffff8113a691>] oom_kill_process+0x201/0x360
Apr 18 14:56:52 ip-172-31-10-191 kernel: [13149385.042861] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 18 14:56:52 ip-172-31-10-191 kernel: [13149385.346663] Out of memory: Kill process 331 (mem) score 932 or sacrifice child
Apr 18 14:56:52 ip-172-31-10-191 kernel: [13149385.349484] Killed process 331 (mem) total-vm:2555698400kB, anon-rss:2379564kB, file-rss:0kB
Apr 18 15:13:41 ip-172-31-10-191 kernel: [13150392.784046] aws invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Apr 18 15:13:41 ip-172-31-10-191 kernel: [13150392.816973] [<ffffffff8113a691>] oom_kill_process+0x201/0x360
Apr 18 15:13:41 ip-172-31-10-191 kernel: [13150392.998954] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Apr 18 15:13:41 ip-172-31-10-191 kernel: [13150393.403518] Out of memory: Kill process 7449 (memB) score 923 or sacrifice child
Apr 18 15:13:41 ip-172-31-10-191 kernel: [13150393.406375] Killed process 7449 (memB) total-vm:14853368kB, anon-rss:7235424kB, file-rss:4kB
See also:
1. time yes | tr \\n x | head -c $((1024*1024*1024*1)) | grep n # Allocated 1 GB 2. https://github.com/julman99/eatmemory/blob/master/eatmemory.c
3. Python
#!/usr/bin/env python
import sys
import time
if len(sys.argv) != 2:
print "usage: fillmem <number-of-megabytes>"
sys.exit()
count = int(sys.argv[1])
megabyte = (0,) * (1024 * 1024 / 8)
data = megabyte * count
while True:
time.sleep(1)
No comments:
Post a Comment