VMware VMKernel lock issue
Start by identifying the server whose VMkernel may be locking the file. To identify the server:
Report the MAC address of the lock holder by running the command (except on an NFS volume):
# vmkfstools -D /vmfs/volumes/UUID/VMDIR/LOCKEDFILE.xxx
Note: Run this command on all commonly locked virtual machine files (as listed at the start of the Solution section) to ensure that all locked files are identified.
For servers prior to ESX 4.1, this command writes the output of the command above to the system's logs.
From ESX 4.1, the output is also displayed on-screen. Included in this output is the MAC address of any host that is locking the .vmdk file. To locate this information, check /var/log/vmkernel .
Look for lines similar to:
Hostname vmkernel: 17:00:38:46.977 cpu1:1033)Lock [type 10c00001 offset 13058048 v 20, hb offset 3499520
Hostname vmkernel: gen 532, mode 1, owner 45feb537-9c52009b-e812- 00137266e200 mtime 1174669462]
Hostname vmkernel: 17:00:38:46.977 cpu1:1033)Addr <4, 136, 2>, gen 19, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
Hostname vmkernel: 17:00:38:46.977 cpu1:1033)len 297795584, nb 142 tbz 0, zla 1, bs 2097152
Hostname vmkernel: 17:00:38:46.977 cpu1:1033)FS3: 132: <END supp167-w2k3-VC a3112729.vswp>
The second line (in bold) displays the MAC address after the word owner. In this example, the MAC address of the Service Console or vswif0 interface of the offending ESX host is 00:13:72:66:E2:00.
After logging into the server, the process maintaining the lock can be analyzed.
In versions of ESX equal or greater than 4.0 U3, 4.1 U1 there is a new field which can identify a read only or multi writer lock owner.
You will see an output similar to:
[root@test-esx1 testvm]# vmkfstools -D test-000008-delta.vmdk
Lock [type 10c00001 offset 45842432 v 33232, hb offset 4116480
gen 2397, mode 2, owner 00000000-00000000-0000- 000000000000 mtime 5436998] <------------------- MAC address of lock owner
RO Owner[0] HB offset 3293184 4f284470-4991d61b-4b28-001a64c335dc <---------------- MAC address of read-only lock owner
Addr <4, 80, 160>, gen 33179, links 1, type reg, flags 0, uid 0, gid 0, mode 100600
len 738242560, nb 353 tbz 0, cow 0, zla 3, bs 2097152
If the command "vmkfstools -D test-000008-delta.vmdk" does not return a a valid MAC address in the top field (returns all zeros). Review the field below it, the RO Owner line below it to see which MAC address owns the read only/multi writer lock on the file. In the above example the offending MAC address is:
00:1A:64:C3:35:DC.
In some cases it is possible that it is a Service Console-based lock, an NFS lock or a lock generated by another system or product that can use or read VMFS file systems. The file is locked by a VMkernel child or cartel world and the offending host running the process/world must be rebooted to clear it.
Once you have identified the host or backup tool (machine that owns the MAC) locking the file, power it off or stop the responsible service, then restart the management agents on the host running the virtual machine
to release the lock.
Report the MAC address of the lock holder by running the command (except on an NFS volume):
# vmkfstools -D /vmfs/volumes/UUID/VMDIR/LOCKEDFILE.xxx
Note: Run this command on all commonly locked virtual machine files (as listed at the start of the Solution section) to ensure that all locked files are identified.
For servers prior to ESX 4.1, this command writes the output of the command above to the system's logs.
From ESX 4.1, the output is also displayed on-screen. Included in this output is the MAC address of any host that is locking the .vmdk file. To locate this information, check /var/log/vmkernel .
Look for lines similar to:
Hostname vmkernel: 17:00:38:46.977 cpu1:1033)Lock [type 10c00001 offset 13058048 v 20, hb offset 3499520
Hostname vmkernel: gen 532, mode 1, owner 45feb537-9c52009b-e812- 00137266e200 mtime 1174669462]
Hostname vmkernel: 17:00:38:46.977 cpu1:1033)Addr <4, 136, 2>, gen 19, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
Hostname vmkernel: 17:00:38:46.977 cpu1:1033)len 297795584, nb 142 tbz 0, zla 1, bs 2097152
Hostname vmkernel: 17:00:38:46.977 cpu1:1033)FS3: 132: <END supp167-w2k3-VC a3112729.vswp>
The second line (in bold) displays the MAC address after the word owner. In this example, the MAC address of the Service Console or vswif0 interface of the offending ESX host is 00:13:72:66:E2:00.
After logging into the server, the process maintaining the lock can be analyzed.
In versions of ESX equal or greater than 4.0 U3, 4.1 U1 there is a new field which can identify a read only or multi writer lock owner.
You will see an output similar to:
[root@test-esx1 testvm]# vmkfstools -D test-000008-delta.vmdk
Lock [type 10c00001 offset 45842432 v 33232, hb offset 4116480
gen 2397, mode 2, owner 00000000-00000000-0000- 000000000000 mtime 5436998] <------------------- MAC address of lock owner
RO Owner[0] HB offset 3293184 4f284470-4991d61b-4b28-001a64c335dc <---------------- MAC address of read-only lock owner
Addr <4, 80, 160>, gen 33179, links 1, type reg, flags 0, uid 0, gid 0, mode 100600
len 738242560, nb 353 tbz 0, cow 0, zla 3, bs 2097152
If the command "vmkfstools -D test-000008-delta.vmdk" does not return a a valid MAC address in the top field (returns all zeros). Review the field below it, the RO Owner line below it to see which MAC address owns the read only/multi writer lock on the file. In the above example the offending MAC address is:
00:1A:64:C3:35:DC.
In some cases it is possible that it is a Service Console-based lock, an NFS lock or a lock generated by another system or product that can use or read VMFS file systems. The file is locked by a VMkernel child or cartel world and the offending host running the process/world must be rebooted to clear it.
Once you have identified the host or backup tool (machine that owns the MAC) locking the file, power it off or stop the responsible service, then restart the management agents on the host running the virtual machine
to release the lock.
Comments
Post a Comment