[WIP] [HyperV2012R2+] Refactor for performance

[berkut1]

Description

This PR is related to issue #224 and aims to optimize Hyper-V performance on high-load nodes. On average, performance will improve by about 2x, and in some cases, up to 6x. On nodes with fewer than 100 servers, there may be no noticeable changes, except for a more responsive control panel when working with VMs.

1. A modern tool, MI/CIM, has been added:

https://learn.microsoft.com/en-us/previous-versions/windows/desktop/wmi_v2/windows-management-infrastructure

Unlike WMI, MI/CIM supports not only DCom but also the modern DSMan:
https://support.quest.com/kb/4311903/comparison-between-dcom-wmi-and-winrm-technologies

By default, MI uses DCom for Remote Hyper-V. This can be changed in the Provider Settings.

2. Multi-command support for PowerShell has been added, allowing multiple commands to run without switching to script mode.

Example PowerShell:

Get-VM -Id 'VM_ID' | Select-Object -Property Id, Name, State

C#

var commands = new List<Command>
{
    new Command("Get-VM")
    {
        Parameters = { { "Id", vmId } }
    },
    new Command("Select-Object") { 
        Parameters = {
            {
                "Property", new string[] { 
                    "Id", 
                    "Name",
                    "State",
                }
            }
        }
    }
};

3. Some methods have been partially optimized.

For example, retrieving all virtual machines (tested with 400 VMs):
Pure Powershell:

Measure-Command {
Get-VM 
}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 26
Milliseconds      : 719
Ticks             : 267193560
TotalDays         : 0.000309251805555556
TotalHours        : 0.00742204333333333
TotalMinutes      : 0.4453226
TotalSeconds      : 26.719356
TotalMilliseconds : 26719.356

WMIv2 imitation (around x20 times faster):

Measure-Command {
Get-CimInstance -Namespace "root\virtualization\v2" -ClassName Msvm_ComputerSystem
}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 393
Ticks             : 13939983
TotalDays         : 1.61342395833333E-05
TotalHours        : 0.00038722175
TotalMinutes      : 0.023233305
TotalSeconds      : 1.3939983
TotalMilliseconds : 1393.9983

Retrieving data for a single server (testing on a high-load node):

Measure-Command {
Get-VM -Name "VM_NAME"
}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 695
Ticks             : 16958672
TotalDays         : 1.96280925925926E-05
TotalHours        : 0.000471074222222222
TotalMinutes      : 0.0282644533333333
TotalSeconds      : 1.6958672
TotalMilliseconds : 1695.8672

WMIv2 (around x2 time faster)

Measure-Command {
Get-CimInstance -Namespace "root\virtualization\v2" -ClassName Msvm_ComputerSystem -Filter "ElementName='VM_NAME'"
}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 683
Ticks             : 6835916
TotalDays         : 7.91193981481481E-06
TotalHours        : 0.000189886555555556
TotalMinutes      : 0.0113931933333333
TotalSeconds      : 0.6835916
TotalMilliseconds : 683.5916

IMPORTANT: When testing, you must start a new PowerShell session each time (open a new PowerShell ISE). Otherwise, after the first run, data will be cached in the current session. Since SolidCP creates a new session for each request, it cannot use cached PowerShell queries. As seen in the results, the first run is so slow in PowerShell that it negates its advantages.

These data simply show how poorly PowerShell performs in some cases. However, I don’t plan to replace everything with WMI because, as always, Microsoft can’t decide which tool should replace another. As a result, PowerShell sometimes has access to data that WMI doesn’t - and vice versa. 🙂
In this PR, I only plan to replace/rewrite the most problematic areas.

[berkut1]

@FuseCP-MTiggelaar @FuseCP-TRobinson @IvanSchulz
Could you test if Remote Hyper-V is still working? As far as I understand, the old WMI uses the DCom protocol (possibly PowerShell does as well) for remote operations. In the new MI, I’ve added a choice between DCom and WSMan, with DCom set as the default.

Unfortunately, I don’t have the ability to test Remote Hyper-V at the moment, so I’d really appreciate your help in deciding whether to continue replacing old WMI methods with the modern MI/CIM or revert to using the old WMI.

For testing, you can simply install/reinstall a test VM server or check memory/hdd VM usage, or just check if "VirtualMachine Thumbnail Image" works (some MI methods are already being used in these areas).

P.S. When building the source code, make sure to switch to the refactor-for-performance branch :)

[IvanSchulz]

Hi, I tried to test it today but I have no experience with remote HV. There are no errors, when I connect to the remote server via FQDN. But when I create the VM I get the following errors in Audit Log:

00:00:00 Setup external network
00:00:00 Assigning external IP addresses
00:00:00 Setting external primary IP address
00:00:00 Setup management network
00:00:00 Management network setup - Skipped
00:00:00 Setup private network
00:00:00 Private network setup - Skipped
00:00:00 Setup DMZ network
00:00:00 DMZ network setup - Skipped
00:00:00 OS Template Generation: 2
00:00:00 SecureBoot: Enabled
00:00:00 OS Template: Win10
00:00:00 Start Converting template VHD
00:00:00 Template VHD path: C:\ClusterStorage\Volume1\SolidCP\Templates\win10.vhdx
00:00:00 Converting to 'C:\ClusterStorage\Volume1\Hyper-V\HyperV1\51\test.h1.loc\Virtual Hard Disks\test.h1.loc.vhdx'
00:00:06 Template VHD size: 40 GB
00:00:06 Expanding to 50 GB
00:00:06 Start Expanding VHD
00:00:29 Error mounting VHD: Server was unable to process request. ---> The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)
Server was unable to process request. ---> The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)
at SolidCP.EnterpriseServer.Code.Virtualization2012.Tasks.CreateVirtualMachineTask.CreateVirtualMachineInternal(String taskId, VirtualMachine vm, LibraryItem osTemplate, Int32 externalAddressesNumber, Boolean randomExternalAddresses, Int32[] externalAddresses, Int32 privateAddressesNumber, Boolean randomPrivateAddresses, String[] privateAddresses, Int32 dmzAddressesNumber, Boolean randomDmzAddresses, String[] dmzAddresses, String summaryLetterEmail) in C:\Users\ischulz\source\repos\v2.x\SolidCP\Sources\SolidCP.EnterpriseServer.Code\Virtualization2012\Tasks\CreateVirtualMachineTask.cs:line 424
00:00:29 VPS was not created

I get the error with both old and new source code, so it's not due to the PR. Do you have an idea?

[berkut1]

@IvanSchulz Thank you very much for trying. But converting a 40GB disk in just 6 seconds doesn't seem very realistic.

If you’re getting the same error in version 1.5.1 (without compiling the code from GitHub), then the issue is related to the Remote HyperV setting.

Did you test it without recompiling the code as well? (Meaning only using the files from the 1.5.1 installer.)

Unfortunately, I’m not familiar with the error "The RPC server is unavailable.", so I can’t suggest anything beyond what I can find online.

If Remote HyperV works with the files installed from the 1.5.1 installer, then you can ignore the text below. :)

Do you have DCom configured? https://help.ivanti.com/ht/help/en_US/IDAC/vNow/setup/setup-DCOM.htm
https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/ecometer-for-uim/OC/knowledgeshare/set-up-wmi-and-dcom_2.html
https://www.aggsoft.com/opc-data-logger/tutorials/dcom-opc-config-windows-8-2012-4.htm Important, it should be configure as impersonate (without password)

[IvanSchulz]

With original 1.5.1 version I got the same error. The only difference is that the original version shows the wrong number of CPU cores.

[berkut1]

Yeah, the original 1.5.1 had a bug with cores, which was fixed here:
28ff929

So if you're still getting that error, that's a good sign that my PR might work. :) The problem is probably in the settings. As far as I know, DCOM requires the firewall to be set up correctly and both servers to be in the same domain, along with some of the settings I mentioned earlier.

If both servers are already in the same domain, you can try disabling the firewall for testing. Probably this can help too - https://learn.microsoft.com/en-us/windows-server/security/rpc-interface-restrict (or this maybe too https://softwarekeep.com/blogs/troubleshooting/fix-the-rpc-server-is-unavailable-error-in-windows )

I’d really appreciate it if you could figure it out. If not, no worries - I’ll just have to request a second test dedicated server to simulate Remote-HyperV. I was just hoping that if someone already has experience with this and has tested Remote Hyper-V, it would save me a lot of time. :)