How does the Linux out of memory (OOM) killer affect a VMware virtual machine running MongoDB?

The Linux OOM killer adds points for processes with high resident set size and high child process memory size. It deducts points for kernel processes and for processes with low nice values. After the killer calculates the points per process, the highest ranking process is killed. 

In a virtual machine, an inflating VMware balloon driver (vmmemctl) can trigger the OOM killer. The balloon driver is likely to become active when the total amount of RAM allocated for all of the VMs that are managed by a hypervisor exceeds the amount of RAM available to the hypervisor itself. An individual VM may have less RAM provisioned than the amount available to the hypervisor, yet the balloon driver can be active.

For example, a hypervisor has 10 Gb of RAM available and two VMs provisioned with 8 and 6 Gb of RAM (=14 Gb total). So long as the actual memory used by the VMs remains below 10 Gb, they will operate normally. However, if the first VM allocates its entire provisioned 8 Gb, and at the same time the second VM is using 3 Gb, the total memory required (11 Gb) is greater than the 10 Gb available to the hypervisor. At this point, the balloon driver starts inflating on the second VM in order to induce the guest OS to reclaim memory from guest applications so the hypervisor can fulfill the memory allocation request made by the first VM. The balloon consumes 4 of the 6 Gb available to the second VM, and the hypervisor uses these 4 Gb to meet the first VM's requirements.

If MongoDB was running on the second VM, this reclaiming process could interfere with MongoDB's memory management, affecting database performance, or might result in the mongod process being killed.

Recommendations

While you can disable the balloon driver to mitigate this issue, this is not recommended as it can cause the hypervisor to use swap space to fulfill memory requests. Accessing data in swap is much slower than accessing data in memory, and can cause performance issues.

We recommend reserving sufficient memory in the hypervisor for MongoDB VMs to ensure the database working set fits within it. 

VMware enables memory compression by default. We recommend disabling memory compression on VMs hosting MongoDB.