Removing Datastores/LUNs within vSphere 5

It’s been a while since my last post (too long!) and I thought I’d talk about something that has recently come back to bite me in the rear end.

I’m sitting at my desk at home doing some work, in particular documenting a procedure for doing some volume migration that needs to happen on several clusters. I’ve been stepping through my documentation to test it and I’ve hit a strange issue which appears to be linked to problems occurring when removing LUNs from a host incorrectly.

I had unmounted several datastores that were no longer required, while still maintaining the recommended two for HA datastore heart-beating. I made sure that I disabled SIOC on the datastores and when un-mounting them I was shown lots of green ticks telling me I’d completed all of the prerequisites.

However, I proceeded to un-present the LUNs from the hosts without first detaching the LUN from the Storage->Devices view:

Unmounting Device

Unmounting Device

Bear in mind that the above screenshot shows attempting to unmount a test volume I have, hence the red tick!

What can happen if you don’t perform this step is an APD or All Paths Down state for the host. Cormac Hogan has a great article here about the APD condition: http://blogs.vmware.com/vsphere/2011/08/all-path-down-apd-handling-in-50.html

Unfortunately for me in this particular case is that I un-presented the LUNs without properly doing an unmount. When I tried a rescan of the iSCSI software HBA the operation eventually timed out and my host disconnected. I now have a host running production VM’s that cannot be managed via the vSphere client or via SSH/Powershell AND the DCUI has frozen! Yay.

So in summary, if you want to ensure you don’t cause yourself any unwanted pain when removing datastores/LUNs from a host or cluster, make sure you follow the KB article PROPERLY! http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004605

Oh, and to add further pain to this problem it seems to have the same symptom that I’ve seen before when applying host profiles and the vmotion vmkernel port ip address gets applied to the management vmkernel port…WEIRD!

Anyway, I’d better get back to fixing what I’ve broken 😛

Advertisements