Issue:
When guest-VMs are being migrated between Hyper-V Hosts within a cluster, CPU core scheduling seems to have been disproportionately distributed toward even-numbered cores (0,2,4,6,8, etc.)
Cause:
There appears to be a new type of processor scheduling named ‘core scheduler’ in Windows Server 2019 that succeeds the previous versions of Windows of ‘classic scheduler’. The difference between those two types of scheduling would affect how migrating VMs would be pinned toward certain numbered CPUs. In the screenshot above, odd-numbered CPU cores seem to be excluded from those migrated VMs.
Resolution:
Check VMHost Supported Versions (notice the IsDefault field):
PS C:\Windows\system32> Get-VMHostSupportedVersion
Name Version IsDefault
---- ------- ---------
Microsoft Windows 8.1/Server 2012 R2 5.0 False
Microsoft Windows 10 1507/Server 2016 Technical Preview 3 6.2 False
Microsoft Windows 10 1511/Server 2016 Technical Preview 4 7.0 False
Microsoft Windows Server 2016 Technical Preview 5 7.1 False
Microsoft Windows 10 Anniversary Update/Server 2016 8.0 False
Microsoft Windows 10 Creators Update 8.1 False
Microsoft Windows 10 Fall Creators Update/Server 1709 8.2 False
Microsoft Windows 10 April 2018 Update/Server 1803 8.3 False
Microsoft Windows 10 October 2018 Update/Server 2019 9.0 True
Check Guest VMs Configured Versions:
PS C:\Windows\system32> Get-VM | FT Name,Version,State
Name Version State
---- ------- -----
TESTVM01 8.0 Off
TESTVM02 8.0 Running
TESTVM03 9.0 Off
TESTVM04 9.0 Running
Perform ‘whole-sale’ Updates:
Note: only ‘offline’ guest VMs will be able to update. Any running VMs will throw errors upon invoking the Update-VMVersion command
PS C:\WINDOWS\system32> Get-VM | Update-VMVersion
Confirm
Are you sure you want to perform this action?
Performing a configuration version update of "TESTVM01" will prevent it from being migrated to or imported on previous
versions of Windows. This operation is not reversible.
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): A
Update-VMVersion : The operation cannot be performed while the virtual machine is in its current state.
At line:1 char:10
+ Get-VM | Update-VMVersion
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (VirtualMachine ...-5cb7879f5c4f']:VirtualMachine) [Update-VMVersion],
VirtualizationException
+ FullyQualifiedErrorId : InvalidState,Microsoft.HyperV.PowerShell.Commands.UpdateVMVersion
Check VM Processor’s Hardware Thread Counts:
PS C:\WINDOWS\system32> Get-VM | Get-VMProcessor | FT VMName,HwThreadCountPerCore
VMName HwThreadCountPerCore
------ --------------------
TESTVM01 1
TESTVM01 1
TESTVM01 1
TESTVM01 1
Fix the CPU Core Scheduling Affinity:
Note: the below command only affects off-lined guest VMs
PS C:\WINDOWS\system32> Get-VM | Set-VMProcessor -HwThreadCountPerCore 0
Set-VMProcessor : Failed to modify device 'Processor'.
Cannot change the processor functionality of a virtual machine now.
'TESTVM02' failed to modify device 'Processor'. (Virtual machine ID C99B-4469-A0E7)
Cannot change the processor functionality of virtual machine 'TESTVM02' while it is running. (Virtual machine ID
C99B-4469-A0E7)
At line:1 char:10
+ Get-VM | Set-VMProcessor -HwThreadCountPerCore 0
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [Set-VMProcessor], VirtualizationException
+ FullyQualifiedErrorId : InvalidState,Microsoft.HyperV.PowerShell.Commands.SetVMProcessor
Generalized Approach:
To fix all guest VMs that have a lowered version of VM Host CPU support, each of the VM must be turned off. If that is not convenient due to production impacts, one could run this command to only target off-lined VMs:
Get-VM |?{$_.Version -lt 9.0}|Update-VMVersion -Force -EA SilentlyContinue|?{$_.State -eq 'Off'}|Set-VMProcessor -HwThreadCountPerCore 0
In our environment, the VM update command has no adverse effects on the performance of the machines thereafter. More importantly, the guest VMs have powered on without errors triggered by these changes. Also, it appears that version 9.0 guest VMs may have been patched to optimally allocate CPU resources. Hence, the Set-VMProcessor -HwThreadCountPerCore 0 may be unnecessary as of this writing. Still, setting that value as zero is recommended for consistency.
Result:
Screenshot of a more balanced CPU Core Scheduling