Load balanced ESXi cluster with Host memory usage alarm

Sometimes a DRS enabled cluster could be in the following situation:

cluster utilization

The cluster is balanced even if three hosts triggered the host memory usage alarm:

cluster status

There is no recommendation, so the DRS is working. How can be three host with high memory usage?

The answer is simple: the graph is showing consumed memory, not active memory. Consumed, is the memory.

  • Consumed Host Memory - Amount of machine memory used on the host. Consumed memory includes Includes memory used by the Service Console, the VMkernel, vSphere services, plus the total consumed metrics for all running virtual machines.
  • Active Guest Memory - Amount of guest “physical” memory actively used.

Let’ see a real time graph:

host memory usageThe server has 128 GB of RAM:

  • Consumed: 125 GB
  • Active: 20 GB

How can this be possibile?

Let’s see a detail of a VM:

VM under utilized

This VM is configured with 12 GB of RAM, but it’s using less than 1 GB. Moving away this VM to another host won’t give enough benefits compared to the latency needed for a vMotion.

What about memory reclamation?

The question now is: why ESXi does not reclaim unused (non-active) memory? From a very Memory reclamation, when and how?:

The high memory state has a threshold hold of 6%, that means that 6% of the ESX host physical memory minus the service console memory must be free. When the virtual machines use less than 94% of the host physical memory, the VMkernel will not reclaim memory because there is no need to, but when the memory usage starts to fall towards the free memory threshold the VMkernel will try to balloon memory. The VMkernel selects the virtual machines with the largest amounts of idle memory (detected by the idle memory tax process) and will ask the virtual machine to select it’s idle memory pages.

In other words:

  • memory usage <= 94%: no action
  • 94% < memory usage <= 96%: memory reclamation using baloon driver (vmmemctl), starting from the VM with the largest amounts of idle memory
  • 96% < memory usage <= 98%: memory reclamation using baloon driver and swapping
  • 98% < memory usage: memory reclamation using swapping.