Memory Tuning of an ESXi host

Memory management and configuration is a huge subject, which can’t be covered in a single post, so the aim here is to cover some of the main configuration options and features relating to memory management, hopefully with some useful examples. I’ll also include some links to some useful documents on memory management and configuration.

Memory Management Techniques

There are a number of methods through which an ESXi host can reduce the amount of physical memory allocated to a virtual machine.

• Page Sharing: ESXi is able to share memory pages between virtual machines, eliminating redundant pages.
• Ballooning: ESXi can use ballooning to force a VM to give up memory pages that the guest OS considers least valuable. VMtools is required as it includes the vmmemctl module which makes ballooning possible. The guest OS must also be configured with sufficient swap space.
• Memory Compression: If there is a danger of host level swapping, then ESXi will use memory compression to reduce the number of pages that it needs to swap out.
• Swap to Host Cache: If compression doesn’t reduce memory usage sufficiently, ESXi will reclaim memory by swapping memory pages to the host cache. Host cache is stored on SSD so is faster than regular swapping (where the files are generally stored on non-SSD devices).
• Regular Swapping: If there is no host cache configured, or it is full ESXi will swap out pages to the virtual machine swap file. If this occurs then there is likely to be severe performance degradation.

Virtual Machine Swap Files

When a virtual machine is powered on, the host creates a swap file for the virtual machine. The size of this swap file is equal to the difference between the virtual machine’s configured memory and the memory reservation for the VM (if one is set).

By default, virtual machine swap files are created in the virtual machine’s working directory (the same location as the .vmx file). However, it is possible to change the default location. The first thing to do is to configure the cluster swap file setting:

Bear in mind, that if you choose to store the swapfiles in a specified datastore, rather than with the virtual machines, then vMotion performance can be degraded. For example, you may choose to store swapfiles on local datastores.

After updating the cluster settings you need to configure the host setting. This is found under the Configuration tab for the host. Click on ‘Virtual Machine Swapfile Location’:

Again, by default the swap files are created in the virtual machines working directory. To configure a specific swapfile location click ‘Edit’:

After choosing the datastore to store swap files on, click ok. This setting is specific to the host, so you will need to make the change on all hosts in the cluster.

Swap file location can be overridden on a per virtual machine basis by setting the option in the virtual machine’s settings:

VMX Swap Files

Along with the swap files detailed above, with ESXi 5 a second swap file is used for every virtual machine (that has a guest hardware version of 7 or above). This swap file is dedicated for memory overhead for the virtual machine and is used when the host is is under memory constraint. You can see this swap file in the data store browser, prefixed with ‘vmx’:

The amount of memory overhead required is determined by a number of different factors including the number of vCPU allocated, the amount of RAM, and whether 3D support is enabled.

Host Cache Configuration

Datastores that are created on SSD storage devices can be used to allocate space for host cache. You can read a lot more about this feature here.

To configure the host cache, go to the host’s configuration tab in the vSphere client and click ‘Host Cache Configuration’. You should see any existing datastores that reside on SSD storage. If you have yet to create a datastore on your SSD storage you can from this pane by clicking ‘Add Storage’.

To configure Host Cache, select a datastore and click properties:

After clicking ok, and refreshing the storage, a bunch of .vswp files will be created on the datastore:

Memory Sharing Configuration

You can use advanced settings to configure the rate that the host scans memory to look for redundant pages. The following two advanced settings can be changed:

• Mem.ShareScanTime
• Mem.ShareScanGHz

You can also disable memory sharing on a per-virtual machine basis by setting the following setting to false:

• sched.mem.pshare.enable

This setting is found under Configuration Parameters in the advanced settings for a virtual machine:

Configuring Memory Compression

Memory compression is enabled by default, however it can be disabled by changing the following advanced setting:

• Mem.MemZipEnable (change to enable/disable memory compression)
• Mem.MemZipMaxPct (change to set the maximum percentage of a vm’s memory that can be compressed – default value is 10)

Memory Tax for Idle Virtual Machines

Idle virtual machine memory can be reclaimed using the balloon driver. The host will identify the virtual machines with the largest amounts of  idle memory and begin to reclaim. You can change the idle memory tax rate using the ‘Mem.IdleTax’ setting.

VMKernel Memory

The ‘Mem.MinFreePct‘ advanced setting is used to set the minimum percentage value for host much host memory we wish to keep free.

Prior to vSphere 5 this value was always set at 6%, however with hosts with a large amount of memory this meant that a lot of RAM was being unnecessarily reserved for VMkernel tasks. For example, for a host with 256 GB ram, 30 GB would be reserved.

With vSphere 5, this value is more of a sliding scale. For example rather than always leaving 6% free, for hosts with 4 – 12 gigabytes of RAM, 4% is kept free for the VMkernel, and for hosts with more 12 GB ram, 2 % of RAM is kept available.

Whatever the value is set as defines when the host begins to reclaim memory using ballooning or swapping. Within the free memory, there are a number of thresholds at which the host will use different methods to reclaim memory. Using figures from vSphere 4 the following will take place:

• 6 percent free (High): Begin ballooning.
• 4 percent free (Soft): Ballooning  and begin compressing virtual memory.
• 2 percent free (Hard): VM swapping.
• 1 percent free (Low): No new pages are provided to virtual machines. 

You can read more about this concept here.

Memory Management Best Practices

VMware highlight a number of best practices in this document, including:

• Do not disable the page sharing or balloon driver.
• Set appropriate reservations.
• Host memory should be larger than guest memory usage.
• Set an appropriate virtual machine memory size.