What do you do when a path to a NFS Datastore goes down?

• Ben Liebowitz
• April 22, 2016April 25, 2016
• 4
• 1317

My QA environment uses Isilon as it’s backend storage platform.  The Isilon share is one large storage volume that is setup with different paths (IPs) to the storage.  This environment has 6 datastores (6 paths).  The other day, we had one of those paths go down, taking down approx 75 VMs with it.

I debated attempting to remove the down datastore and then add it back with the same name (with a different IP) but didn’t know what that would do to the VMs, if they’d come back cleanly or not, etc.  In the end, I decided to go down a different path…

While my storage engineer troubleshot the issue on their side, I decided to spin up another datastore, using a different IP.  I then modified my script to migrate VMs between vCenters to pull a list of VMs on that specific datastore, power the VMs off, remove them from inventory on the down datastore, then re-add them back on the newly created one and power them back on.  I left the “update-tools” line in there too as many of the VMs were out of date on the tools version.  A few extra minutes of downtime in a QA environment won’t hurt anything. 

You’ll notice a section of the script where it pauses…  I did this so I could then edit the migrate.csv and change the datastore name.  This was easier than taking the time to modify the code to hardcode the new datastore name.  However, if you want to modify line 109 with the actual datastore name and the pause can be removed.  The modified line is here:

$vmxfile = "[NEW_DATASTORE_NAME] $($vmName)/$($vmName).vmx"

Again, props to Luc Dekens (@LucD22 / www.lucd.info) for the folder path functions!

Next step is to use DNS entries instead of IPs.  This way, if this happens again, it’s just a matter of changing the DNS record and waiting for replication. 🙂

###################################################################
#
# Script to migrate VMs from down Isilon datastore to the new one
# Created by BLiebowitz on 4/21/2016
#
###################################################################

# Set Variables
$vcenter = "vcenter_server"
$esxhost = "vSpherehost.domain.com"
$datastore_old = "Datastore03"
$datastore_new = "Datastore04"

# Load PowerCLI Module
if ( !(Get-Module -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue) ) {
. “C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1”
}

# Connect to vCenter
connect-viserver $vcenter

#Build the BlueFolderPath function
New-VIProperty -Name 'FullPath' -ObjectType 'VirtualMachine' -Value {
param($vm)

$current = Get-View $vm.ExtensionData.Parent
$path = ""
do {
$parent = $current
if($parent.Name -ne "vm"){$path =  $parent.Name + "/" + $path}
$current = Get-View $current.Parent
} while ($current.Parent -ne $null)
$path.TrimEnd('/')
} -Force | Out-Null

#Build the Get-folderbypath function
function Get-FolderByPath {
<#
.SYNOPSIS  Retrieve folders by giving a path
.DESCRIPTION The function will retrieve a folder by it's
path. The path can contain any type of leave (folder or
datacenter).
.NOTES  Author:  Luc Dekens
.PARAMETER Path
The path to the folder.
This is a required parameter.
.PARAMETER Separator
The character that is used to separate the leaves in the
path. The default is '/'
.EXAMPLE
PS> Get-FolderByPath -Path "Folder1/Datacenter/Folder2"
.EXAMPLE
PS> Get-FolderByPath -Path "Folder1>Folder2" -Separator '>'
#>

param(
[CmdletBinding()]
[parameter(Mandatory = $true)]
[System.String[]]${Path},
[char]${Separator} = '/'
)

process{
if((Get-PowerCLIConfiguration).DefaultVIServerMode -eq "Multiple"){
$vcs = $defaultVIServers
}
else{
$vcs = $defaultVIServers[0]
}

foreach($vc in $vcs){
foreach($strPath in $Path){
$root = Get-Folder -Name Datacenters -Server $vc
$strPath.TrimStart($Separator).Split($Separator) | %{
$root = Get-Inventory -Name $_ -Location $root -Server $vc -NoRecursion
if((Get-Inventory -Location $root -NoRecursion | Select -ExpandProperty Name) -contains "vm"){
$root = Get-Inventory -Name "vm" -Location $root -Server $vc -NoRecursion
}
}
$root | where {$_ -is [VMware.VimAutomation.ViCore.Impl.V1.Inventory.FolderImpl]}|%{
Get-Folder -Name $_.Name -Location $root.Parent -Server $vc
}
}
}
}
}

# export list of VMs to Migrate to CSV
Get-Datastore -Name $datastore_old | Get-VM |
Select Name, @{N="Datastore";E={Get-Datastore -VM $_ | Select -ExpandProperty Name}},
@{N="Network";E={Get-VirtualPortgroup -VM $_ | Select -ExpandProperty Name}},
@{N="Parent";E={$_.FullPath}} |
Export-Csv "c:\migrate.csv" -NoTypeinformation -UseCulture

# Shutdown VMs (cleanly)
foreach ($row in (Import-csv c:\migrate.csv))
{
shutdown-vmguest -vm $row.Name -Confirm:$false
start-sleep -s 60
stop-vm -vm $row.Name -Confirm:$false
}

# Pause script while CSV is edited to change Old Datastore name to New Datastore name.
<span style="color: #454545;">$HOST.UI.RawUI.ReadKey(“NoEcho,IncludeKeyDown”) | OUT-NULL</span><br style="color: #454545;" /><span style="color: #454545;">$HOST.UI.RawUI.Flushinputbuffer()</span>

## Add backto inventory
foreach ($row in (Import-Csv c:\Migrate.csv)) {
$vmName = $row.Name
$vmxfile = "[$($row.Datastore)] $($vmName)/$($vmName).vmx"
$vmNewVM = New-VM -VMFilePath $vmxfile -VMhost $esxhost

# Move VM to existing folder locaiton
Move-VM -VM $vmName -Destination (get-folderbypath $row.Parent)

# Poweron VMs
Start-VM -VM $vmName -Confirm:$false | wait-tools
Get-VMQuestion -VM $vmName | Set-VMQuestion –Option "I moved it" -confirm:$false

# Wait for VM to boot before updating VMware Tools
start-sleep -s 60

# Update VMware Tools
update-tools $vmName

} ## end foreach

Thanks,

Ben Liebowitz, VCP, vExpert
NJ VMUG Leader