HP releases Storevirtual Lefthand OS 12.5

Just this morning I noticed that Lefthand OS 12.5 was now available for the HP Storevirtual platform. I was surprised by this as I hadn’t seen any announcement about it.

Anyway, taking a look at the release notes the minor update has the following new enhancements:

  • Two-node quorum
  • Support for iSCSI split network
  • Support for VSA on RHEL 6.6 and CentOS 6.6
  • MEM driver for vSphere 6.0
  • SCVMM 2012 R2 support

Additionally there are a number of bug fixes.

See here for the full release notes: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=c04735172&lang=en-nz&cc=nz

The most interesting enhancement is the Two-node quorum which I’m hoping to test shortly. This is great news for people wanting to deploy VSA for ROBO environments and is a timely release with the upcoming announcement of VSAN 6.1 at VMworld 2015 which is rumored to bring metro clustering (something Storevirtual already supports).

Anyway, stay tuned for further updates on Lefthand OS 12.5!

Advisory: HP Proliant BL465c G7 – Storage Fault with P410 removed

I posted an issue I found with the HP BL465c G7 blades last year where we were getting storage errors in the iLO 3. These blades were ordered without the P410 controller as they were booting ESXi from SD card and using SAN attached storage for the virtual machines. The iLO would report the storage status strangely (see my earlier post: http://wp.me/p1J73N-41) as it would flip-flop between OK and failed.

After several months of logging a job with HP and being told that a fix would come in iLO 3 v1.6 I eventually tested the new firmware and to my disappointment the issue persisted. I didn’t want my hosts continuously alerting in vCenter with storage failures so I’ve kept my blades at an older version that doesn’t pass this failure through to the CIM.

Anyway, just this morning I received an email from HP providing an advisory link to the problem saying that there is still no firmware fix and as a workaround the SAS/SATA cable should be removed from the backplane.

http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_ba847bafb2a2d782fcbb0710b053ce01=wsrp-navigationalState%3DdocId%253Demr_na-c03961287-1%257CdocLocale%253D%257CcalledBy%253D&javax.portlet.tpst=ba847bafb2a2d782fcbb0710b053ce01&ac.admitted=1382047973743.876444892.492883150

It is certainly not an ideal workaround but would allow me to upgrade the blades to the latest versions of iLO firmware. I’ll look at testing one and post an update on the issue🙂

HP Storevirtual LefthandOS 11.0

Just this morning while browsing the Storevirtual CMC updates FTP share I noticed an announcement file detailing the new LefthandOS 11.0.

Anyway, it is due for release in October according to the blurb and will come with the following new features:

  • Sub-volume auto-tiering with Adaptive Optimization (Exclusive feature for HP StoreVirtual VSA)
  • Support for Microsoft System Center Virtual Machine Manager
  • Smarter Centralized Management Console and Online Upgrades
  • Easier configuration and more granular control of application-managed snapshots with Application Aware Snapshot Manager for Microsoft and VMware environments

The first thing the caught my eye was the auto-tiering! This would be a great addition to the VSA product and from what I have found on the internet it seems that the auto-tiering will be automated and not rely on a schedule like the 3PAR gear. Sadly at this stage the feature won’t apply to existing physical models which I can understand why. It would be nice to breathe some life into them though!

Also the updates to the CMC will be greatly welcomed. Lets hope that it works better than the previous iterations and fingers crossed that it brings in full support for internet proxy authentication!!!

Anyway, when it’s released i’ll test the upgrade and post about the experience🙂

vCenter Converter Standalone – Slow conversion rates and SSL

I thought I’d quickly write about this as it was something I was not previously aware of, mostly due to the fact that I have not performed many P2V migrations using the Converter tool and when I have it’s mostly been with the old offline converter.

Anyway, I was doing some bench-marking of the conversion process on a Windows 2003 R2 server and I couldn’t understand why I seemed to be hitting a network throughput ceiling of around 10MB/s. At first I though it must have been a routing issue as in this particular environment I was using had multiple VLANs and the source machine was in a different VLAN to the ESXi host. The router in this case only had a routing throughput of 100Mb/s so the 10MB/s made me think that this was the case.

However, when I moved the host into the same VLAN as the source machine I got the same speeds…now I was really confused. Everything else in between seemed fine and I could not work out what was making the conversion so slow.

So I jumped on Twitter to see if anyone else had come across this before (admittedly Google probably would have told me :-P) and a couple of smart guys i know @vStorage and @dmanconi suggested I turn off SSL.

I immediately went back to Converter and was clicking around like crazy and thought “where the hell do I turn that off!!??!!??”. Thankfully Google stepped in here and lead me to this VMware KB article: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2020517

Aha! So as of vCenter Converter 5.0 it enables SSL by default…I’m not sure why to be honest, in my opinion security of the traffic during a P2V would be the last thing on my mind, but that’s just me🙂

Anyway, I followed the instructions and set the <useSsl> parameter to False, restarted the Worker service and kicked off my conversion again.

WOW, now I was getting around 50MB/s throughput on my conversion, around five times faster than before! This now meant that my upcoming P2V jobs were going to complete in way less time than I first though.

So a word of advice if you are thinking about doing some P2V’s or are not blown away by how slow they are running, apply this tweak and you will be away laughing.

Oh, while we are on the topic, for those of you using HP Proliant servers, Guillermo Musumeci has written a handy tool for automating the removal of the HP Proliant Support Pack drivers and software after you’ve done your P2V: http://www.ctxadmtools.com/DownloadTool.aspx?ProdID=4cc60321-c738-4a7b-b93d-d0a57244ff31 …unfortunately at this moment the website won’t let me register, but I’m sure it will be working again soon. The tool has been around for some time but is one of those really hand tools to have and saves manually removing the HP drivers and services.

VMware VCAP5-DCA Experience

This morning I received a much anticipated email from VMware Certification. It had been almost three weeks since re-sitting my DCA exam and I was extremely nervous about the result.

My first attempt at the exam was terrible and it really threw me. At the time the lab environment was very slow and I panicked, wasting valuable time just thinking about the time! After that I was totally put off the experience and put my study on hold. In fairness I had just had another baby so I cut myself a little slack🙂

Anyway, three months went by and I geared up once more to attack the beast. I sat the exam at the same testing centre in sunny Tauranga, New Zealand (seems to be the ONLY place you can actually book the exam in NZ anyways). This time the lab environment seemed much faster at responding and I found myself getting through questions far quicker. To be fair I had also put more study into the areas that I had clearly lacked from the first attempt and this made a lot of difference.

The exam was still extremely tough and it was clearly a very different set of questions and lab setup from the first time. I managed to run out of time with about five questions left completely unanswered. I still felt very nervous about how I did but hoped that I had done enough to pass, in this case more than 300/500. My first attempt came in at about 250/500 but I had only answered about 60% of the questions, so I was hopeful that this time was much better.

So, this morning I was up at about 6am with my two boys, half asleep sitting on the couch. I’m on my phone using the web browser to view webmail (don’t ask!) and the attached pdf result wouldn’t download!!!! Then when I finally managed to get it to download it wouldn’t open the pdf, argh!!!!!! I finally got the thing open and scrolled down to the score…389/500, woohoo!

I was so stoked, I really didn’t want to have to sit it a third time! Now that I have both DCD and DCA I can now apply for VCDX. This will be my next major goal and realistically will take me some time. Between a busy job and family life there ain’t a lot of time left for preparing a design, but lets hope that through work I can prepare one suitable.

Finally, to give my two cents worth of advice to others out there considering taking the exam;

1. Work through the blueprint end to end, and I really do mean end to end. The exam covers stuff from all over the blueprint (go figure :-P)

2. Check out Jason Langer and Josh Coen’s study guides here: http://www.valcolabs.com/vcap5-dca/ and here: http://www.virtuallanger.com/vcap-dca-5/, both of these guys fricken rule! Many thanks to them for their massive efforts in creating such a great resource.

3. Use Autolab for creating your home lab environment, it will save you a heap of time: http://www.labguides.com/. A BIG thanks to Alastair Cooke for his work and others that have helped him. It is a fantastic tool for deploying vSphere at home.

4. Make sure that you don’t skip over areas that you think you already know. I did this both times and realised afterwards that I didn’t really know as much as I thought I did!

5. During the exam, manage your time very carefully. Don’t stall on any one particular question too long and if you get stuck, move on and come back later.

6. Oh, and lastly, if you do fail, don’t beat yourself up. It’s a real tough exam with a lot of content to work through in a short space of time. I was way too hard on myself the first time and this put me off getting back in the drivers seat for a long time. Sometimes it is good to fail and gives us perspective.

We’ll that’s about it from me, I’m over the moon about passing and I can sit back for a little while now…just not too long eh! VCDX…

Study hard and good luck!

vSphere Update Manager – Remediation failures due to removable media

This is something that has caught me out a few times before so I thought I’d quickly post about it.

Have you have ever tried to remediate a cluster using VUM and encounter a message something like this?:

vum - error

The message itself doesn’t tell a lot, leaving you to dive into the error logs within vCenter. This can be quite tedious particularly if you don’t know what to look out for. The best place to start is searching for entries containing “Error”…surprise surprise😛

In this particular scenario I have several VM’s with removable media attached to them, bear in mind your particular scenario could be different and may have nothing to do with removable media!

Digging through the VUM logs (normally found under C:\Users\All Users\VMware\VMware Update Manager\Logs or C:\Documents and Settings\All Users\Application Data\VMware\Update Manager) we come across a file named vmware-vum-server-log4cpp.log.

Browsing through the file searching for any errors we find the following entry:

vum - log error

Aha! We now have an explanation to why the remediation failed! You might think the easiest way to fix this is to check each of the VM’s in the cluster and remove any removable media devices…

While it is probably a good idea to remove these generally speaking there may be valid reasons for having them attached. Hence why VUM has a great feature found under “Maintenance Mode Options”!

BEHOLD!

vum - remediate disable media

 

By ticking the box at the bottom the remediation task will automatically disable any attached removable media devices allowing the task to complete successfully.

Anyway, I hope this little trick helps others out there who are wondering why their cluster remediations aren’t completing!

CPU Ready and vCPU over-subscription

I’ve been doing a bit of performance tuning on some of the clusters I look after at work and started looking deeper into CPU Ready times. This particular metric has always been something I’m aware of and it’s impact on performance but I had never gone looking for issues relating to it. Mostly because I’d never had a host or cluster that was that over-subscribed!

Anyway, I thought I’d do a quick post on what CPU Ready times mean, how you can measure them and how you can help reduce them…here goes.

What is CPU Ready???

The term CPU Ready is a bit confusing at first as one would assume that it refers to how much CPU is ready to be used, but this is not the case. The lower the CPU Ready time the better!

CPU Ready is the percent of time a process is ready to run but is waiting for the CPU scheduler to allow that process to run on a physical processor. I.e. “I’m ready to go but I can’t do anything yet!”.

So now that we have a better understanding of what CPU Ready is, lets look at what can cause this value to increase and hurt your VM’s performance.

What causes CPU Ready times to increase???

1. Over-commitment/Over-subscription of physical CPU

This would be the most common cause and can happen when you have committed too many vCPU’s in relation to the number of physical CPU cores in your host.

From what I have read it seems that for best performance you should keep your pCPU:vCPU ratio equal to or less than 1:3. So in other words, if your host has a total of four CPU cores, you should not allocate more than a total of 12 vCPU to the VM’s on that host. This isn’t to say you can’t have more but you may run into performance problems doing so.

2. Using CPU affinity rules

Using CPU affinity rules across multiple VM’s can cause high CPU Ready times as this can restrict how the CPU scheduler balances load. Unless specifically required I would not recommend using CPU affinity rules.

3. Using CPU limits on virtual machines

Another potential cause of CPU Ready is using CPU limits on virtual machines. Again, from what I have read I would suggest that you do not use CPU limits unless absolutely necessary. CPU limits can prevent the scheduler from allocating CPU time to a VM if it were to violate the limit set, hence causing ready times to increase.

4. When Fault Tolerance is configured on a VM

The last scenario could be where you have deployed a VM using FT and the primary and secondary VM can’t keep up with the synchronisation of changes. When this happens the CPU can be throttled causing higher ready times.

Now that we’ve covered what can cause CPU Ready times to increase, lets look at how to measure them and reduce them. For this example I’ve used the most common cause, over-provisioning.

How do I look for CPU Ready issues???

Take the example below; it is a VM that has been configured with four vCPU. Looking at the last days CPU usage you can see this particular VM is doing almost nothing (it is a test VM).

Image

When I then look at the CPU Ready times for the same period I see that the summation value is around 9200ms. Remember that both of these charts are the last day roll up.Image

Now you are probably thinking, what the hell does that mean? Well, we can convert this summation into a percentage to make things a little easier to quantify.

The formula is simply this:

CPU Ready % = (Summation value / (chart update interval in seconds x 1000)) x 100

Each of the available update intervals are listed below (refer to KB article 2002181: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2002181)

  • Realtime: 20 seconds
  • Past Day: 5 minutes (300 seconds)
  • Past Week: 30 minutes (1800 seconds)
  • Past Month: 2 hours (7200 seconds)
  • Past Year: 1 day (86400 seconds)

So returning to our previous chart of the last day we get:

(9200 / (300 x 1000)) x 100 = 3.06%

Now this isn’t a bad CPU Ready time percentage but it will do for the purposes of this example. VMware recommends that for best performance CPU Ready % should be less than 5%.

Based on the fact that my virtual machine is by no means busy and it has been given 4 vCPU I will now drop this back to 2. Yes I could drop it back even further to 1 but for the purposes of this example I’ll bring it back to 2🙂

After powering off the VM, changing the vCPU and powering back on I get a significant drop in CPU Ready time as seen below in a Real-time chart.

Image

Running a new calculation on this value of around 54ms (a very rough guesstimate average :-P) we get this:

(54 / (20 x 1000)) x 100 = 0.27%

As you can see the average CPU Ready time has decreased quite significantly by simply lowering committed resource to the VM. Obviously this would only be practical on VM’s that are not vCPU constrained.

In my experience most people (myself included) over allocate vCPU, particularly when translating vendor hardware requirements into virtual machine requirements! Some of the worst I’ve seen are when sizing some of the Microsoft System Center products. The sizing guides often suggest dual quad-core physical servers, but this does not mean you should give your VM eight vCPU.

I think the best approach is to size lower and adjust accordingly if you are hitting a CPU resource limit. Spend some time looking over your environment and see where you might be able to tune your performance, you might be surprised at how much you can improve it!

Follow

Get every new post delivered to your Inbox.

Join 215 other followers