vSAN 6.6 Rebalance & Resync Operations

In this post I will delve a little bit into day 2 OPS here with vSAN. Suppose you have had a maintenance window for rebooting and patching hosts, or perhaps you have a message under vSAN health saying that a proactive disk rebalance is needed on your vSAN cluster. In either scenario you will have components of VM's resyncing or in the case of a rebalance, moving on the cluster. This won't be a long post, but will give you some idea of what to expect when you encounter either disk resync or proactive rebalancing on your vSAN cluster.

First off let's define what the difference between the two operations are. A resync is replicating VM components across hosts in accordance to Storage Policy-Based Management or SPBM FTT=1, FTT=2 etc.

Updating a vSAN Cluster using VUM

I wanted to shed some light on what some may think is a mystery or difficult task. How do I update a vSAN cluster using vSphere Update Manager (VUM)? The quick and simple answer is easily! The great thing about vSAN is that you already know how to upgrade and patch it since it is built into the ESXi hosts and vCenter web client that you already use daily. I will take you through the process of upgrading a host in my home lab. Remember as with any upgrade make sure that your hardware is supported and on the VMware Hardware Compatibility List (HCL) before proceeding with ANY upgrade.

The first step is to make sure you have a baseline attached to your cluster and/or Hosts. You can navigate to the update manager tab in the vCenter Web Client to do this. A baseline is a group that you setup for either patches, or upgrades and then you assign it to physical ESXi hosts or Clusters. You can see here that we have baselines created and VUM tells us that this particular host that we have highlighted is not compliant with this baseline and needs to be patched in order to be in compliance. We will go ahead and create a baseline for this host so you can see that process.

For this example I am going to create a baseline for critical host patches. I will check that box when creating the baseline, in this case since the critical host patches is a predefined baseline I don't need to go any further to assign to my hosts, another way it can be done is by right clicking a cluster or a host and then choosing to attach a baseline that way as well.

Here you can see the current patch level on the host is ESXi 6.5u1 build 5969303 in its non compliant state.

Now that we have created a baseline and told VUM to scan for updates and can see that we are not in compliance, we can start the process of upgrading. What I like to do first is enter the host into maintenance mode, VUM will do this for you but I prefer to do this step manually. 

As we enter maintenance mode we are presented with a choice on what to do with the vSAN data that resides on this host, while in maintenance mode. We can evacuate all data to other hosts in the cluster, a very nice addition in vSAN 6.6 is the ability to see how much data would need to be moved. Per VMware recommendations there is no need to do a full migration for only a patch, that will take hours, or days depending on what your data is like on the vSAN array. Ensure accessibility is sufficient in this case as the host will only be down for a few moments. If it is down for longer than that due to some sort of issue, be it hardware or software, depending on your SPBM and your vSAN node design, you should at least have 1 copy of the data and after 60 minutes if a host doesn't come back up vSAN will self heal back to full redundancy on the remaining hosts (provided you have at least a 4 node RAID 1 setup required for self healing or 5 nodes in RAID5)

Now that we have our host set to go into maintenance mode, migrate all VM's off of the host with vMotion, once the host has fully gone into maintenance mode you can click the remediate button to start the patching process. You will select your baseline as the critical host patches. In my case the same patch was also listed as a vSAN recommended patch as well. Click Next.

Here we can see how many patches this baseline will apply, this will be only 1 patch here to get to the latest version. Click Next.

Here we can see the actual patch name to be applied, what product the patch is for, and its actual release date.

On this next step we see that we have options to run this patch install as a scheduled task. We also see an ignore warnings box. I have always run my patches right away as I want to keep an eye on everything and make sure things come back up OK.

You can choose your VM options on this screen, but again the way I do this to make sure that ALL VM's are vMotioned off before I get to this point simply for more control, but this screen is where the host will vMotion VM's off the host and go into Maintenance mode if you choose to do it this way. In a vSAN environment I would recommend you do these options manually as well. It is great that the system can do it, I just prefer seeing each step through. Also on each one of these coming screens you will see the option to save the parameters you choose on that page as the default host remediation options going forward, so you do not have to choose them every time. This is a handy time saving feature I would recommend taking advantage of.

We see more options for the cluster level on this screen and we have to turn off DPM during the remediation as well as turn off FT, and HA admission control, if you do not do it in this step you will get a warning of it being on in a later step and it will prevent the remediation process from going any further. 

We are almost ready to hit finish, but at the top there is another step I always do, just as a precaution and that is to run the Pre-Check wizard, it will warn you if a service is on that will prevent Remediation from occurring.

Here is what the pre-check report looks like with HA admission control still turned on in the cluster, it will warn you that it is on and then it will give you a recommendation on what action to take to rectify the situation so that you can continue with the patching. 

From here on out the process is automatic. You will see the host do what is essentially a live image boot onto the new ESXi version and then it will download the installation script from Update Manager and install the new version. It will take a while for the host to do this and then reboot a second time where it will boot from the installed version. You can now see the version shows as build 6765664. It will take a while for vSAN to reinitialize the SSD's in my case and then will finish booting.

At this point the host will show back up in vCenter. You can now exit maintenance mode and the cluster will go back to normal functionality. You can go to the monitor tab and click on vSAN and then the Resyncing Components section and you will notice that the vSAN disks will be resyncing data that changed on the cluster disks while this host was offline. It shouldn't take long for this to complete, but you can watch the progress here and it will show you which components from VM's that are resyncing and how much data is left to resync from each component.

We can go back to the vCenter Web Client Update Manager page and run a Scan for Updates check and see that it is now compliant with the patch baseline that it wasn't compliant with prior to the patch cycle. One other thing, I also go back and make sure to turn back on the HA admission control that was turned off on the cluster during the remediation. I hope that helps answer some questions on the process and showed how easy it is to maintain patch levels on your vSAN clusters. Until next time. Cheers!

Setting up Encryption in vSAN 6.6

I have had this post in the works for a while now. I have had some challenges in my home lab with unstable storage and issues with fiber channel support on my freenas server. I have since moved over to a IBM DS3400 SAN and that has been much more stable, anyway I digress, onto the content!

I wanted to play around with the new encryption options that vSAN 6.6 brings to the table and I was fortunate enough to get setup with a HyTrust KMS server so I can get it up and running and test it out. I want to give a huge shoutout to the HyTrust team for providing this to me, without their support this post would not be possible! I will not be doing any benchmarks but rather seeing how the technology works and how easy it is to implement.

The first step is to have a working vSAN cluster. I happen to have a 6 node nested ESXI 6.5u1 vSAN cluster in my lab with each nested host containing 2 vCPU 16GB of RAM, 1 10GB Cache disk and 1 50GB capacity disk for a total of 300GB on the vSAN Datastore. This Cluster is running FTT=2. I will be installing the Hytrust KMS server into the nested environment.

VMware Licensing Organization Tips

I bet there are lots of folks out there that just love managing licensing both on their systems and in the VMware portal. For those of you that didn't sense my sarcasm, there have been some pain points in the past, but I have to say I have been so impressed with recent changes to the VMware portal and some additions in vSphere 6.5 regarding license management that I had to share once I found them.

They have made it really easy to use and have separated the product lines in the main licensing page, this is done by default and is not something you have to setup. The portal is laid out really nice and is pretty easy to navigate compared to how things have been in the past.

Another really cool feature is the ability to split and combine CPU licenses really easy.

VSAN 6.2 Deployment Part 3 Implementation

Today we will be getting back to the VSAN deployment series I had started earlier this summer, it's time to continue on with excerpts from the presentation I have been doing at VMUG's on my VSAN deployment. Today we will be focusing on the Implementation phase. Here is an overview of that process.

We started by getting our 4 new Lenovo servers racked then installed esxi 6.0u2 on them, we then installed our 10Gbe switches and configured our VLANS separating out our VSAN traffic on its own VLAN which we also isolated from talking to the rest of the network as we were told that the multicast traffic VSAN 6.2 required could get quite chatty and could take down other network switches on the network (note that this requirement was removed in VSAN 6.6 which now uses unicast) Our network topology in part 2 of this blog series showed our dual redundancy we planned for by having 2 10Gbe switches and 2 PCIE 10GBe interface cards per host.

VMware Community Involvement

I have been very blessed to have many mentors that have given me sound advice in my career. They have seen my desire to learn, and given me advice on how to grow via skill set and other means. Mentors have helped to continue the learning process to help me grow my career. One of those means has been getting involved in VMware and VMUG communities.

Last week I am honored to share that I was awarded the VMware vExpert VSAN award for my contributions to the community around the vSAN product. Thank you to Corey Romero and his team for all they do to support this community! I have to say this wouldn't be possible without those mentors! They gave me the push I needed to apply for the program and succeed. I have done a lot of promotion of this product through twitter, doing VMUG presentations at local meetings and USERCON events throughout the Midwest. These presentations were based upon my production install experience with VSAN. This will be a short post but I wanted to give some mentoring advice of my own based on my experiences.

VMworld 2017 Preview

It's hard to believe that it's almost here already, but VMworld 2017 is just around the corner! It is a very exciting time for those in the virtualization community. There are so many great learning opportunities, with very good speakers on some great topics such as VSAN, VMware on AWS, NSX and many more. I thought I would put together a short post with some of the sessions I plan on attending while I am out there.


My Session List

vSAN Vision: The Future of HCI [STO1490SU]

vSAN in VMC on AWS environment [VMTN6656U]

What's New in vSAN 6.6: Technical Deep Dive [STO2047BU]