While creating/consolidating snapshots it was failing with: message = “Consolidation failed for disk node ‘scsi0:1’: 12 (Cannot allocate memory).”

When I looked into the logs found that VMFS-6 heap memory exhaustion.

vmkernel.log:

2020-08-31T 00:31:23.413Z cpu21:5630454)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 4ac136e7-g3tyyq21-4477-5f67cz28e370: Out of memory
2020-08-31T 00:31:23.413Z cpu21:5630454)Vol8: 4202: Failed to get object 28 type 1 uuid 4ac136e7-g3tyyq21-4477-5f67cz28e370 FD 0 gen 0 :Out of memory
2020-08-31T 00:31:23.413Z cpu21:5630454)Vol8: 4202: Failed to get object 28 type 2 uuid 4ac136e7-g3tyyq21-4477-5f67cz28e370 FD 4 gen 1 :Out of memory
2020-08-31T 00:31:23.413Z cpu21:5630454)Vol8: 4202: Failed to get object 28 type 2 uuid 4ac136e7-g3tyyq21-4477-5f67cz28e370 FD 4 gen 1 :Out of memory

vmkwarning.log:

vmkwarning.0:2020-06-16T13:28:23.291Z cpu48:3479102)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.
vmkwarning.0:2020-06-16T14:20:23.676Z cpu62:3479103)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.
vmkwarning.0:2020-06-17T08:05:54.057Z cpu92:3909762)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.

To workaround the issue the memory allocated gets freed during VMFS resource allocation and hence creating a thick file on VMFS datastore.

Follow the steps below on each ESXi host for each datastore:

  1. Create Eager zeroed thick disk on all of the mounted VMFS6 datastores.
vmkfstools -c 10M -d eagerzeroedthick /vmfs/volumes/datastore/eztDisk
  1. Delete the Eager zeroed thick disk created in step 1.
vmkfstools -U /vmfs/volumes/datastore/eztDisk

Happy Learning and keep sharing..!! 🙂

Advertisement