Intro:
There was this situation when we experienced connection issues with “Connect-VIServer” commands from PowerShell and went on a ghost hunt to chase down the root cause.
Set VMWare PowerCLI to ignore cert errors
PS C:\WINDOWS> Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$False
Scope ProxyPolicy DefaultVIServerMode InvalidCertificateAction DisplayDeprecationWarnings WebOperationTimeout
Seconds
----- ----------- ------------------- ------------------------ -------------------------- -------------------
Session UseSystemProxy Multiple Ignore True 300
User Ignore
AllUsers
Attempted to connect to vSphere via port 443
PS C:\WINDOWS> connect-viserver vcenter.kimconnect.com -Protocol https
connect-viserver : 5/23/2019 11:19:44 AM Connect-VIServer Error: Invalid server certificate. Use
Set-PowerCLIConfiguration to set the value for the InvalidCertificateAction option to Prompt if you'd like to connect
once or to add a permanent exception for this server.
Additional Information: Could not establish secure channel for SSL/TLS with authority 'vcenter5.kimconnect.com'.
At line:1 char:1
+ connect-viserver vcenter55.kimconnect.com -Protocol https
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : SecurityError: (:) [Connect-VIServer], ViSecurityNegotiationException
+ FullyQualifiedErrorId : Client20_ConnectivityServiceImpl_Reconnect_CertificateError,VMware.VimAutomation.ViCore.
Cmdlets.Commands.ConnectVIServer
Check Local Certificate Store for VCENTER certs
PS C:\WINDOWS> Get-ChildItem -Path Cert:\LocalMachine\ -recurse | Select-String "vcenter"
<Empty Result>
Ran this Script to check TLS Connectivity. It would do 2 things:
- Set local machine to use the proxy for external domain and bypass proxy for internal URIs
- Attempt to obtain the Public Key certificate from a provided URI. In this case, we’re checking the VCENTER.kimconnect.com machine running vSphere 5.5
$proxy="http://proxy:8080";
$exclusionList="localhost;*.kimconnect.com"
function checkProxy{
try{
$connectionTest=iwr download.microsoft.com
#$connectionSucceeds=Test-NetConnection -Computername download.microsoft.com -Port 443 -InformationLevel Quiet
if ($connectionTest){
$GLOBAL:haveInternet=$True;
return $True;
}
}
catch{
return $False
}
}
function fixProxy{
# Check if proxy is enabled on the system and fix it
$proxyKey=(Get-ItemProperty -Path "Registry::HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings")
if ($proxyKey.ProxyEnable){
# Set http proxy for browsers
Set-Itemproperty -path "Registry::HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings" -Name 'ProxyServer' -value $proxy
# Set winhttp proxy for PowerShell
netsh winhttp set proxy $proxy $exclusionList
[system.net.webrequest]::defaultwebproxy = New-Object system.net.webproxy($proxy)
[system.net.webrequest]::defaultwebproxy.credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
[system.net.webrequest]::defaultwebproxy.BypassProxyOnLocal = $true
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
}
if (checkProxy){
"Proxy is now good to go..."
$GLOBAL:haveInternet=$True;
}
else{
"Proxy problems..."
$GLOBAL:haveInternet=$False;
break;
}
}
function getPublicKey{
[OutputType([byte[]])]
PARAM (
[Uri]$Uri
)
if ($uri.Scheme -eq $null){
$uri="https://"+$uri;
}
else {
if (!($uri.Scheme -eq "https")){
$uri="https://"+$uri.Authority;
}
}
[Net.ServicePointManager]::Expect100Continue = $true;
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
[Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
[Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
$request = Invoke-WebRequest -Uri $uri
#$request = [System.Net.HttpWebRequest]::Create($uri)
try{
#Make the request but ignore (dispose it) the response, since we only care about the service point
$request.GetResponse().Dispose()
}
catch [System.Net.WebException]{
if ($_.Exception.Status -eq [System.Net.WebExceptionStatus]::TrustFailure){
#We ignore trust failures, since we only want the certificate, and the service point is still populated at this point
}
else{
#Let other exceptions bubble up, or write-error the exception and return from this method
throw
}
}
#The ServicePoint object should now contain the Certificate for the site.
$servicePoint = $request.ServicePoint
$key = $servicePoint.Certificate.GetPublicKey()
Write-Output $key
}
fixproxy;
getPublicKey vcenter.kimconnect.com;
Output if TLS 1.0 or TLS 1.1 was enabled
PS C:\Users\kimconnect\Desktop\> C:\Users\kimconnect\Desktop\unit-test.ps1
Current WinHTTP proxy settings:
Proxy Server(s) : http://proxy:8080
Bypass List : localhost;*.kimconnect.com
Proxy is now good to go...
Exception calling "GetResponse" with "0" argument(s): "The underlying connection was closed: An unexpected error occurred on a receive."
At C:\Users\kdoan\Desktop\Notes\unit-test.ps1:70 char:9
+ $request.GetResponse().Dispose()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : WebException
Output if TLS 1.2 was enabled
Invoke-WebRequest : The request was aborted: Could not create SSL/TLS secure channel.
At C:\Users\kimconnect\Desktop\unit-test.ps1:66 char:16
+ $request = Invoke-WebRequest -Uri $uri
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
You cannot call a method on a null-valued expression.
At C:\Users\kimconnect\Desktop\unit-test.ps1:72 char:9
+ $request.GetResponse().Dispose()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
You cannot call a method on a null-valued expression.
At C:\Users\kimconnect\Desktop\unit-test.ps1:86 char:5
+ $key = $servicePoint.Certificate.GetPublicKey()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
Check Server IIS
$ssl3=Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server"
$tls=Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server"
$tls11=Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Server"
$tls12=Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Server"
$protocols = @($ssl3,$tls,$tls11,$tls12)
$key1="DisabledByDefault"
$key2="Enabled"
foreach ($p in $protocols){
$pName=Split-Path $p.PSParentPath -Leaf;
$pValueInHex='0x{0:x}' -f $p.$key2;
$pName+" "+$key2+" value: "+$pValueInHex
}
Sample server output:
SSL 3.0 Enabled value: 0x0
TLS 1.0 Enabled value: 0xffffffff
TLS 1.1 Enabled value: 0xffffffff
TLS 1.2 Enabled value: 0xffffffff
Check Client Protocols
$ssl3=Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Client"
$tls=Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client"
$tls11=Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client"
$tls12=Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client"
$protocols = @($ssl3,$tls,$tls11,$tls12)
$key1="DisabledByDefault"
$key2="Enabled"
foreach ($p in $protocols){
$pName=Split-Path $p.PSParentPath -Leaf;
$pValueInHex='0x{0:x}' -f $p.$key2;
$pName+" "+$key2+" value: "+$pValueInHex
}
Sample client output:
SSL 3.0 Enabled value: 0x0
TLS 1.0 Enabled value: 0x0
TLS 1.1 Enabled value: 0x0
TLS 1.2 Enabled value: 0xffffffff
Interpretation of the server and client’s outputs:
It appears that the server SSL protocols are including TLS, TLS 1.0, and TLS 1.2. However, the admin’s workstation (running Windows 10) would only connect using TLS 1.2. The common denominator between client and server is TLS 1.2. That explains the Invoke-Webrequest results prior, where TLS 1.2 connection attempts would yield the error of “Could not create SSL/TLS secure channel,” instead of “The underlying connection was closed: An unexpected error occurred on a receive” when using TLS 1.0 & 1.1.
Enable TLS 1.1 Client
$tls11Path="Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client"
$key2="Enabled"
Set-Itemproperty -path $tls11Path -Name $key2 -Type DWord -Value 0xffffffff
Get-ItemProperty -path $tls11Path
Disable TLS 1.1 Client
$tls11Path="Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client"
$key2="Enabled"
Set-Itemproperty -path $tls11Path -Name $key2 -Type DWord -Value 0x0
Get-ItemProperty -path $tls11Path
Further Troubleshooting Progress:
Although the issue would most likely caused by the server’s certificate, that cannot be quickly remedied without collecting more data points about this issue. Once sufficient information has been gathered, a proposal to change the server’s SSL certificate would have a higher chance of administrative approval. Hence, the following vague attempts are good sources for memorandums.
a. Regenerate vCenter SSL certificates with minimum public key length of 2048 bits
b. Uninstall Microsoft(Windows) update 3175024
https://knowledge.broadcom.com/external/article?legacyId=2146002
Modify this registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\KeyExchangeAlgorithms\Diffie-Hellman]”ClientMinKeyBitLength”=dword:00000200
$path="Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\KeyExchangeAlgorithms\Diffie-Hellman"
$key="ClientMinKeyBitLength"
$value=512
Set-Itemproperty -path $path -Name $key -value $value -Type DWord
#Set-Itemproperty -path $path -Name "ServerMinKeyBitLength" -value 2048
Get-ItemProperty -Path $path
https://support.microsoft.com/en-us/topic/sha512-is-disabled-in-windows-when-you-use-tls-1-2-5863e74e-e5b6-cc3b-759b-ece8da875825
PS Cert:\LocalMachine> $path1="Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\00010003"
PS Cert:\LocalMachine> (Get-ItemProperty -Path $path1 -Name "Functions").Functions
RSA/SHA256
RSA/SHA384
RSA/SHA1
ECDSA/SHA256
ECDSA/SHA384
ECDSA/SHA1
DSA/SHA1
RSA/SHA512
ECDSA/SHA512
Check for installed KBs
dism /online /get-packages | findstr KB3175024
dism /online /get-packages | findstr KB3172605
dism /online /get-packages | findstr KB3161608
Run PowerShell as Administrator
cd "C:\Program Files\VMware\CIS\vSphereTlsReconfigurator\VcTlsReconfigurator"
.\reconfigureVc backup -d S:\backup
This error occurred
VMWARE_CIS_HOME is not defined
This was supposed to the the command to disable TLS 1.0 & 1.1 (which had not been successfully ran)
.\reconfigureVc update -p TLSv1.2
Root Cause
After exhausting the available options of technical troubleshooting, the solution was simply removing the multiple versions of vSphere clients being installed on the problematic workstation. Knowing the cause, it would be possible to further investigate by checking those programs. Since vSphere client was a closed-source application, the buck ended at this correlation. It was unfortunate since correlation should not be construed as causation, but oh well…
Categories: