Issues updating VMware Tools with Update Manager

• Matt Liebowitz
• July 26, 2010
• 4
• 1910

I recently updated a lab environment to vSphere 4 Update 2 and wanted to upgrade VMware Tools on several VMs at once.  I decided to use Update Manager to push out the VMware Tools update, and during the process discovered an issue between Update Manager and newer versions of Windows.  I didn’t try any of the following with vSphere 4.1.

I remediated a folder of about seven VMs and to my surprise I saw that only a few of them actually updated.  The others simply sat there and never updated the tools, rebooted, or did much of anything.  Sorting the folder by Guest OS I noticed something – all of the VMs that did not update were running either Windows Server 2008 or Windows 7.

In doing some testing I found that if I initiated the VMware Tools upgrade from the vSphere Client it would mount and run setup correctly, but if initiated from Update Manager it would do nothing.  The problem is caused by a Windows service called Interactive Services Detection, explained nicely in this blog post on MSDN.  The short version is this – starting in Windows Vista, services run in session 0 while all user sessions run in session 1.  If a service tries to interact with the desktop (in session 0), Interactive Services Detection will freeze the application and present the user with the following dialog:

In the case of the VMware Tools install you have to actually be logged into the VM to see the message.  If you select “View the message” you are brought to the desktop of session 0 and will then be able to proceed with the VMware Tools install.  Unfortunately this only pops up automatically if you are using a 32-bit OS.  If you are using a 64-bit OS, such as Server 2008 R2, you will need to manually start the Interactive Services Detection service to see the above dialog.  If you do nothing at all, the VM will sit there perpetually waiting for you to allow the installer to proceed.

vCenter will continue to wait for the VMware Tools to finish installing, so it leaves the VM in a state where many operations can’t be performed.  Things like reverting a snapshot, trying to use the VMware Tools to shut down a VM, etc., results in the following:

VMware Tools Install still in progress:

Trying to shutdown/reboot the VM from vCenter:

The only way I found to break it of that cycle (other than clicking the “View this message” option and proceeding with the install) is to reboot the virtual machine.  That essentially ends the VMware Tools install and returns the VM to a normal state.  Not even selecting “End VMware Tools Install” will actually break it out of the install.

I found a relatively simple way around this issue – set your VMware Tools service to start as a user with rights to the server instead of Local System.  This allows the install to occur in Session 1 (a user session) and the install proceeds as expected.  I imagine that most do not run VMware Tools as a regular user but instead as the default Local System account so this isn’t an easy fix.  It requires using a tool like Group Policy Preferences to globally change the account that the service runs as (which won’t take effect until a reboot), and also updating templates so it isn’t an issue for new VMs.

This is probably more of a combination of a Microsoft issue and a VMware issue.  Perhaps VMware can change the way Update Manager initiates the VMware Tools upgrade in the future.  And although not pretty the above does work as a workaround until it is resolved.