I upgraded my Windows 2008 Hyper-V host server to Windows 2008 R2 last night. The upgrade went smoothly, but when it settled down as the new version, I found two virtual machines out of seventeen to be missing from the Hyper-V console UI.
From the Hyper-V event log i see
Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin
Source: Microsoft-Windows-Hyper-V-VMMS
Date: 6/4/2011 2:31:26 AM
Event ID: 16300
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: elune
Description:
Cannot load a virtual machine configuration: General access denied error (0x80070005) (Virtual machine ID 5185AC13-4148-4AFE-9024-6E74FE3C9754)
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Hyper-V-VMMS" Guid="{6066F867-7CA1-4418-85FD-36E3F9C0600C}" />
<EventID>16300</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2011-04-05T18:31:26.432942100Z" />
<EventRecordID>641</EventRecordID>
<Correlation />
<Execution ProcessID="1964" ThreadID="2064" />
<Channel>Microsoft-Windows-Hyper-V-VMMS-Admin</Channel>
<Computer>elune</Computer>
<Security UserID="S-1-5-18" />
</System>
<UserData>
<VmlEventLog xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events" xmlns="http://www.microsoft.com/Windows/Virtualization/Events">
<VmName>
</VmName>
<VmId>5185AC13-4148-4AFE-9024-6E74FE3C9754</VmId>
<ErrorMessage>%%2147942405</ErrorMessage>
<ErrorCode>0x80070005</ErrorCode>
</VmlEventLog>
</UserData>
</Event>
These "missing" VMs do not have any online snapshots tied to them. Even for those that did I removed and merge the snapshots based on Microsoft's advice, and they've "survived".
There seem to be a lot of "General access denied error (0x80070005)" discussion when it comes to starting virtual machines. But my case is the Hyper-V service being unable to even load the configuration, so Hyper-V knows where those VM config locations but don't have permission to access them?
The three Hyper-V services are launched with LOCAL SYSTEM account, and the "Virtual Machines" folders for those VMs do grant Full Control permissions. What I observe from the other VMs is their folders have additional ACEs for Virtual Machines group and the VM GUID itself?
I tried to duplicate this ACE structure, but Windows cannot locate the GUID principals of those missing VMs. What would be causing this problem?
Well, this is a strange means.
The key thing is Windows/Hyper-V is referring to a "list" somewhere to tell it what virtual machines are registered with the server. My virtual machines are scattered across multiple drives, so there has to be a centralised repository. Which turns out to be
C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines
there are symbolic links that refer to the physical locations of the actual config XML files. I noticed the troubled VM links had a "lock" icon.
I edited the security ACL of the troubled symbolic link. Sure enough it does not have the NT Virtual Machine account like the rest, so instead I granted it Full Control to the Users group. Restarted Hyper-V Virtual Machine Management service, and it could once again load up the missing VMs. It appears to work without the Virtual machine accounts.
Still haven't gotten the complete deep answers I was looking for to explain just what Hyper-V is requiring out of those Virtual machine accounts, but at least the original virtual machine config can be reused.