HP releases Storevirtual Lefthand OS 12.5

Just this morning I noticed that Lefthand OS 12.5 was now available for the HP Storevirtual platform. I was surprised by this as I hadn’t seen any announcement about it.

Anyway, taking a look at the release notes the minor update has the following new enhancements:

  • Two-node quorum
  • Support for iSCSI split network
  • Support for VSA on RHEL 6.6 and CentOS 6.6
  • MEM driver for vSphere 6.0
  • SCVMM 2012 R2 support

Additionally there are a number of bug fixes.

See here for the full release notes: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=c04735172&lang=en-nz&cc=nz

The most interesting enhancement is the Two-node quorum which I’m hoping to test shortly. This is great news for people wanting to deploy VSA for ROBO environments and is a timely release with the upcoming announcement of VSAN 6.1 at VMworld 2015 which is rumored to bring metro clustering (something Storevirtual already supports).

Anyway, stay tuned for further updates on Lefthand OS 12.5!

Advisory: HP Proliant BL465c G7 – Storage Fault with P410 removed

I posted an issue I found with the HP BL465c G7 blades last year where we were getting storage errors in the iLO 3. These blades were ordered without the P410 controller as they were booting ESXi from SD card and using SAN attached storage for the virtual machines. The iLO would report the storage status strangely (see my earlier post: http://wp.me/p1J73N-41) as it would flip-flop between OK and failed.

After several months of logging a job with HP and being told that a fix would come in iLO 3 v1.6 I eventually tested the new firmware and to my disappointment the issue persisted. I didn’t want my hosts continuously alerting in vCenter with storage failures so I’ve kept my blades at an older version that doesn’t pass this failure through to the CIM.

Anyway, just this morning I received an email from HP providing an advisory link to the problem saying that there is still no firmware fix and as a workaround the SAS/SATA cable should be removed from the backplane.


It is certainly not an ideal workaround but would allow me to upgrade the blades to the latest versions of iLO firmware. I’ll look at testing one and post an update on the issue 🙂

HP Storevirtual LefthandOS 11.0

Just this morning while browsing the Storevirtual CMC updates FTP share I noticed an announcement file detailing the new LefthandOS 11.0.

Anyway, it is due for release in October according to the blurb and will come with the following new features:

  • Sub-volume auto-tiering with Adaptive Optimization (Exclusive feature for HP StoreVirtual VSA)
  • Support for Microsoft System Center Virtual Machine Manager
  • Smarter Centralized Management Console and Online Upgrades
  • Easier configuration and more granular control of application-managed snapshots with Application Aware Snapshot Manager for Microsoft and VMware environments

The first thing the caught my eye was the auto-tiering! This would be a great addition to the VSA product and from what I have found on the internet it seems that the auto-tiering will be automated and not rely on a schedule like the 3PAR gear. Sadly at this stage the feature won’t apply to existing physical models which I can understand why. It would be nice to breathe some life into them though!

Also the updates to the CMC will be greatly welcomed. Lets hope that it works better than the previous iterations and fingers crossed that it brings in full support for internet proxy authentication!!!

Anyway, when it’s released i’ll test the upgrade and post about the experience 🙂

vCenter Converter Standalone – Slow conversion rates and SSL

I thought I’d quickly write about this as it was something I was not previously aware of, mostly due to the fact that I have not performed many P2V migrations using the Converter tool and when I have it’s mostly been with the old offline converter.

Anyway, I was doing some bench-marking of the conversion process on a Windows 2003 R2 server and I couldn’t understand why I seemed to be hitting a network throughput ceiling of around 10MB/s. At first I though it must have been a routing issue as in this particular environment I was using had multiple VLANs and the source machine was in a different VLAN to the ESXi host. The router in this case only had a routing throughput of 100Mb/s so the 10MB/s made me think that this was the case.

However, when I moved the host into the same VLAN as the source machine I got the same speeds…now I was really confused. Everything else in between seemed fine and I could not work out what was making the conversion so slow.

So I jumped on Twitter to see if anyone else had come across this before (admittedly Google probably would have told me :-P) and a couple of smart guys i know @vStorage and @dmanconi suggested I turn off SSL.

I immediately went back to Converter and was clicking around like crazy and thought “where the hell do I turn that off!!??!!??”. Thankfully Google stepped in here and lead me to this VMware KB article: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2020517

Aha! So as of vCenter Converter 5.0 it enables SSL by default…I’m not sure why to be honest, in my opinion security of the traffic during a P2V would be the last thing on my mind, but that’s just me 🙂

Anyway, I followed the instructions and set the <useSsl> parameter to False, restarted the Worker service and kicked off my conversion again.

WOW, now I was getting around 50MB/s throughput on my conversion, around five times faster than before! This now meant that my upcoming P2V jobs were going to complete in way less time than I first though.

So a word of advice if you are thinking about doing some P2V’s or are not blown away by how slow they are running, apply this tweak and you will be away laughing.

Oh, while we are on the topic, for those of you using HP Proliant servers, Guillermo Musumeci has written a handy tool for automating the removal of the HP Proliant Support Pack drivers and software after you’ve done your P2V: http://www.ctxadmtools.com/DownloadTool.aspx?ProdID=4cc60321-c738-4a7b-b93d-d0a57244ff31 …unfortunately at this moment the website won’t let me register, but I’m sure it will be working again soon. The tool has been around for some time but is one of those really hand tools to have and saves manually removing the HP drivers and services.

Storage alert bug running vSphere on HP BL465c G7 blades

I have recently been configuring some BL465c G7 blades at work running vSphere 5.0 Update 1 installed on an internal SD card and using FC storage for the datastores. I ran into a strange issue where from within the vSphere client my blade hosts would show as having alerts after running HP’s October firmware bundle containing the BIOS version A19 15/8/12.

Hosts showing alert status

Hosts showing alert status

After some investigation I found that they had storage faults, specifically the following alarm:

Alert Detail

Alert Detail

Then when drilling into the Hardware Status tab (with the HP offline bundle installed) it shows that the storage controller has drive faults or missing disks as seen below.

Storage status showing failed drives

Storage status showing failed drives

This was really weird as these particular blades were ordered without the integrated P410 controller as we weren’t planning on using local disk. Weird…

Anyway, I spent a while trying different things to clear the alerts without any luck. Diving into the iLO from the System Information summary, the storage tab shows the same error:

iLO storage summary

iLO storage summary – BAD

But after a few minutes and a page refresh, the status clears and looks as expected:

iLO storage summary - OK

iLO storage summary – OK

This continues to flip-flop between good and bad every minute or so. WHAT???

I decided to roll back the BIOS and iLO firmware (as well as the iLO which made no difference) and what was interesting is that when going back to BIOS version A19 19/3/12, the iLO status still shows the same problem as above, but the CIM provider information within vSphere no longer shows the storage controller…because it DOESN’T HAVE ONE! :-). See the screenshot below:

Older BIOS doesn't present storage controller CIM information

Older BIOS doesn’t present storage controller CIM information

I logged a job with HP and after working through several diagnostics with them they came to the conclusion that this was definitely a bug and would be addressed in a future BIOS update.

So for those of you out there that have been pulling your hair out like I have, there is a bug and there is no immediate fix other than rolling back to BIOS rev. A19 19/3/12 or earlier. Either that or you have to live with your hosts continually showing alerts 🙂 NOTE: Rolling back the BIOS only masks the alert from coming through into the vSphere client and still shows up in the iLO status summary. However, the alert doesn’t appear to generate any IML event logs…it also does not show up in HP SIM either.

This bug only appears to affect blades that don’t have the optional P410 controller installed and I only have BL465c G7 blades to test this on. It may also affect BL465/460c G7 and earlier where the controller is optional.

UPDATE: I have been told by HP that the bug is caused by the disk drive backplane being active even when the controller isn’t present and they also suggest that it can be observed with any BIOS/iLO combination. I have also found that some blades seem fine with just the BIOS rollback while some still bring the storage controller status back into vSphere. For these odd few a rollback to iLO2 1.28 seems to fix the problem, hence I am making this my baseline for now.

UPDATE – October 2 2013

After stumbling across a few updates to the iLO 3 firmware I noticed that v1.57 specifically mentions the following fix:

  • Incorrect Drive status shown in the iLO 3 GUI when the HP Smart Array P410 controller is removed from the ProLiant c-Class Blade BL465c G7.

However, after testing this new firmware the same problem exists and is also present on the latest v1.61 firmware. What is interesting to see is that the error is slightly different in that while the drives flip-flop between “not installed” and “Fault”, the number of drive bays does not change now. Now the number of drives is always correctly shown as two…I guess progress is progress right??? 😛

I’ll open a new case with HP and hopefully find a fix for this hugely annoying bug!!!

HP Lefthand OS 10.0 upgrade issue with certain management group password strings

A few weeks ago HP released the new Lefthand OS 10.0 for their StoreVirtual products. I’ve since upgraded my VSA clusters successfully,  however, when attempting this on a test P4300 cluster I ran into some weird issues.

During the upgrade (which now has a nice progress window) the first node in the cluster would upgrade OK but upon rebooting would not reconnect to the CMC. When trying to log into the direct console on the upgraded node it would not accept my Management Group (MG) password. I had no other option but to forcibly cancel the upgrade and I rebuilt the failed node.

After attempting a subsequent upgrade, the same problem occurred. This time I noticed in the status bar in the CMC that it was failing to log into the newly upgraded node and said the username and password was incorrect! As a result I logged a job with HP.

The response I received was rather interesting…as it turns out they have had several incidences similar and are related to MG passwords containing the characters ‘~‘ or ‘$‘.

So, this got me thinking…if the password is corrupted by the upgrade, what happens if I try logging in with a partial password? I tried several combinations on my test cluster until VIOLA, I could log in with the first part of the password leading up until the special character!

To clarify this, lets say our MG password was abc123$def, after the upgrade the CMC will fail to reconnect. You should be able to log into the node using abc123, i.e. we’ve dropped the $ symbol and anything after that. I’m not sure what happens if your password begins with those characters…could be interesting as the CMC does not allow a blank password!

I would imagine HP will release a patch for this in the coming weeks, in the meantime you could change your MG passwords prior to the upgrade 🙂
Read more of this post

HP LeftHand CMC 10.0 Changes

HP’s Lefthand / P4000/ StoreVirtual product has had a major version upgrade with it’s announcement of Lefthand OS 10.0. This release will be the first to drop the SAN/iQ moniker in favor of the company name that created the product before HP’s aquisition a few years ago.

The release of this software upgrade was slated for the 4th December if I’m not mistaken but interestingly their FTP site now has the updated patches/upgrades as of the 26th of November.

I had the chance to download the new files (with some difficulty, I get the feeling their FTP site is taking a hammering at the moment!) and have since installed the new version of their Centralised Management Console or CMC.

Going into this upgrade I had high hopes for its new support for an internet proxy for the downloading of patches, something that has really let the product down previously in my opinion. In any case, the new version now allows you to specify a SOCKS proxy…yay!

Now, the bad news…

It does not allow you to specify any authentication for the proxy…argh!!!! In our environment this is a real pain from a security perspective and as such is not going to help. For now it will be back to downloading the media from an alternative location and copying it to the CMC. This in itself can prove to be tedious, particularly when the CMC decides that the downloaded media is corrupt and needs to re-download it! Oh well…baby steps eh 😛

CMC 10.0 Proxy Setting

On a more positive note, the new version now supports ActiveDirectory integrated authentication. So far I can’t see where this is configured but I’m guessing you’ll need to have your nodes upgraded to version 10 first…i’ll post an update on this shortly.

Further to this there is now an additional SAN status page panel showing all active tasks which should prove to be extremely useful, something that was lacking previously, especially when managing multiple clusters from a single CMC by more than one administrator. Again I’ll post more on this when I see it in action. In the meantime here’s a shot of the Active Tasks pane, not very exciting but gives you an idea.

CMC 10.0 Active Tasks

So that seems to be about it for now, I’d be keen to hear from any others that have found more new features that I’ve missed. Once I’ve fully downloaded all of the new patches I’ll upgrade one of my test VSA clusters and post about that, hopefully I’ll then be able to integrate the cluster security into AD 🙂

Thanks for reading!