Update: FastCopy doesn’t properly update NTFS permissions of parent directories nor files if those items have broken ACLs (e.g. editing root directory using ‘get-acl $source | set-acl $destination’ cmdlet) – preventing FastCopy from accessing them on repeated runs. Therefore, it is not a recommended tool for mission critical data migrations. Emcopy or Robocopy (version XP026 or higher) is recommended, instead.
<# dataSync_v0.0.3.ps1
Purpose: this PowerShell Script is to efficiently mirror large batches of files using FastCopy & Emcopy in conjunction with Volume Shadow Services.
Main Switches:
1. FastSync: Trigger Fastcopy to quickly populate containers at Destinations
2. NormalSync: Use VSS Snapshots & Emcopy
3. FinalSync: disable and disconnect all SMB sessions at the source to quiescent files before running a copy operation to ensure zero data loss.
Use VSS Snapshots, re-copy mismatched files (exclude newer timestamps), send notification email
4. bypassFileCompare: File compare, in this context, means matching hash signatures of each source file with its destination. Thus, this is a CPU-intensive process
that will require a very wide 'cutover time window.' Hence, for directories with millions of files, it is advisable to set this value to $false
so that the program will skip the filecompare routine.
Current Features:
0. Validate the sources and destinations (array/object) and remove invalid items
1. If Antivirus is not blocking PowerShell, copy open files as well as resting files using VSS. This is a MUST to capture all contents.
Create a snapshot of each source volume(s) using Shadow Copy to capture any locked files
VSS automation softwares such as ShadowSpawn have been considered. Unfortunately, such app would introduce complications with RamDisk instantiation.
Be advised that VSS will only capture file contents at a monent-in-time during script execution.
Minor 'data loss' can still occur because files that have the same time stamp and length may still have slightly different contents.
Hence, it is recommended that a 'finalSync' be performed prior to 'cutover' execution.
2. Elevate runtime execution with appropriate system privileges - Execute in the context of an Administrator
3. Initial sync will utilize FastCopy (GPLv3 licensing) as the data shipping engine.
FastCopy is acclaimed to be "the fastest file copying software" on Windows, and it had won a Grand Prize in the Windows Forest Awards on Christmas 2015.
This is to provide consistency as the most popular engine named RoboCopy would vary between different versions of Windows.
Other copying engines such as emcopy, richcopy, gsrichcopy, etc. are worthy considerations. Unfortunately, dataSync requires GPL licensing to be freely available.
__Deprecated__ (i like this big word) utilizing secondary hashing engine named JackSum (GPLv3 licensing), a cyclic redudancy checking engine with a modified version of crc32 (using 8-bit)
to favor speed over probability of heuristic collisions (which is 2^16, assuming R as as a true random seed).
4. Emcopy is the main engine for normalSync and finalSync as that software has more robust reporting and ACLs corrections
5. Detect whether the given source path is a local path or UNC path
If local, then trigger VSS and translate source paths.
If remote, then invoke the same function on the remote machine
6. Output a log of the file copy operation to include stats of items being processed
7. Multi-threading file compare process count to 8 (optimal)
Limitations:
1. This iteration requires that script is triggered from a local Windows machine with Internet access (no proxies) to download fastCopy & Emcopy.
2. Sources must be LFS to work with VSS and Destinations could either be LFS or UNC paths.
#>
# Set the behavior of script in this section
$finalSync=$false # This value overrides the one below
$normalSync=$False
$fastSync=$true # If both values above are false, then sync-type will be 'fastSync'
$bypassFileCompare=$True # Set this to true for large directories so that the program knows to skip fileCompare
# Specify Sources (LFS) and Destinations (UNC)
# Using System.Array Object[] constructor to create a two dimensional Array
$arr=@();
$arr+=[PSCustomObject]@{Clustername='';From='D:\Test';To='\\FILESERVER\Test'}
$arr+=[PSCustomObject]@{Clustername='';From='D:\Test2';To='\\FILESERVER\Test2'}
# SendMail Variables
$sendFrom = "[email protected]"
$sendTo = '[email protected]' #,'[email protected]','[email protected]'
$smtpServer = 'relay.kimconnect.com'
# Autogen variables
$hostname=$ENV:computername
$dateStamp = Get-Date -Format "yyyy-MM-dd-hhmmss"
$scriptName=$MyInvocation.MyCommand.Path
$scriptPath=Split-Path -Path $scriptName
$logPath="$scriptPath`\fileSyncLogs\$hostname"
$snapShotPath='C:\vssSnapshot'
# Sanitize inputs
$syncMode=if($finalSync){'finalSync';
}elseif($normalSync){
'normalSync'
}elseif($fastSync){
'fastSync'
}else{
'normalSync'
}
################################## Excuting Program as an Administrator ####################################
# Get the ID and security principal of the current user account
$myWindowsID=[System.Security.Principal.WindowsIdentity]::GetCurrent()
$myWindowsPrincipal=new-object System.Security.Principal.WindowsPrincipal($myWindowsID)
# Get the security principal for the Administrator role
$adminRole=[System.Security.Principal.WindowsBuiltInRole]::Administrator
# Check to see if we are currently running "as Administrator"
if ($myWindowsPrincipal.IsInRole($adminRole))
{
# We are running "as Administrator" - so change the title and background color to indicate this
$Host.UI.RawUI.WindowTitle = $myInvocation.MyCommand.Definition + "(Elevated)"
#$Host.UI.RawUI.BackgroundColor = "Black"
clear-host
}else{
# We are not running "as Administrator" - so relaunch as administrator
# Create a new process object that starts PowerShell
$newProcess = new-object System.Diagnostics.ProcessStartInfo "PowerShell";
# Specify the current script path and name as a parameter
$newProcess.Arguments = $myInvocation.MyCommand.Definition;
# Indicate that the process should be elevated
$newProcess.Verb = "runas";
# Start the new process
[System.Diagnostics.Process]::Start($newProcess);
# Exit from the current, unelevated, process
exit
}
Write-Host -NoNewLine "Running as Administrator..."
################################## Excuting Program as an Administrator ####################################
function dataSync{
param(
[string]$syncMode,
[Array]$sourceAndDestinationArray,
[bool]$bypassFileCompare,
[string]$snapshotPath,
[String]$logPath
)
$logFile=if($syncMode -eq 'finalSync'){
"$logPath`\FINALSYNC-$dateStamp.txt";
}elseif ($syncMode -eq 'fastSync'){
"$logPath`\FastSync-$dateStamp.txt";
}else{
"$logPath`\NormalSync-$dateStamp.txt";
}
$errorsLogFile="$logPath`\$hostname-filecopy-errors-$dateStamp.txt"
$log="/LOG+:$logFile"
$lockedFilesReport="$logPath`\$hostname-locked-files-log-$dateStamp.txt"
$pathErrorsLog="$logPath`\$hostname-path-errors-log-$dateStamp.txt"
New-Item -ItemType Directory -Force -Path $logpath;
write-host "Executing copying tasks..."
################################## Add System Backup Privileges ####################################
function addSystemPrivilege{
param(
[String[]]$privileges=@("SeBackupPrivilege","SeRestorePrivilege")
)
function includeSystemPrivileges{
$win32api = @'
using System;
using System.Runtime.InteropServices;
namespace SystemPrivilege.Win32API
{
[StructLayout(LayoutKind.Sequential)]
public struct LUID
{
public UInt32 LowPart;
public Int32 HighPart;
}
[StructLayout(LayoutKind.Sequential)]
public struct LUID_AND_ATTRIBUTES
{
public LUID Luid;
public UInt32 Attributes;
}
[StructLayout(LayoutKind.Sequential)]
public struct TOKEN_PRIVILEGES
{
public UInt32 PrivilegeCount;
public LUID Luid;
public UInt32 Attributes;
}
public class Privileges
{
public const UInt32 DELETE = 0x00010000;
public const UInt32 READ_CONTROL = 0x00020000;
public const UInt32 WRITE_DAC = 0x00040000;
public const UInt32 WRITE_OWNER = 0x00080000;
public const UInt32 SYNCHRONIZE = 0x00100000;
public const UInt32 STANDARD_RIGHTS_ALL = (
READ_CONTROL |
WRITE_OWNER |
WRITE_DAC |
DELETE |
SYNCHRONIZE
);
public const UInt32 STANDARD_RIGHTS_REQUIRED = 0x000F0000u;
public const UInt32 STANDARD_RIGHTS_READ = 0x00020000u;
public const UInt32 SE_PRIVILEGE_ENABLED_BY_DEFAULT = 0x00000001u;
public const UInt32 SE_PRIVILEGE_ENABLED = 0x00000002u;
public const UInt32 SE_PRIVILEGE_REMOVED = 0x00000004u;
public const UInt32 SE_PRIVILEGE_USED_FOR_ACCESS = 0x80000000u;
public const UInt32 TOKEN_QUERY = 0x00000008;
public const UInt32 TOKEN_ADJUST_PRIVILEGES = 0x00000020;
public const UInt32 TOKEN_ASSIGN_PRIMARY = 0x00000001u;
public const UInt32 TOKEN_DUPLICATE = 0x00000002u;
public const UInt32 TOKEN_IMPERSONATE = 0x00000004u;
public const UInt32 TOKEN_QUERY_SOURCE = 0x00000010u;
public const UInt32 TOKEN_ADJUST_GROUPS = 0x00000040u;
public const UInt32 TOKEN_ADJUST_DEFAULT = 0x00000080u;
public const UInt32 TOKEN_ADJUST_SESSIONID = 0x00000100u;
public const UInt32 TOKEN_READ = (
STANDARD_RIGHTS_READ |
TOKEN_QUERY
);
public const UInt32 TOKEN_ALL_ACCESS = (
STANDARD_RIGHTS_REQUIRED |
TOKEN_ASSIGN_PRIMARY |
TOKEN_DUPLICATE |
TOKEN_IMPERSONATE |
TOKEN_QUERY |
TOKEN_QUERY_SOURCE |
TOKEN_ADJUST_PRIVILEGES |
TOKEN_ADJUST_GROUPS |
TOKEN_ADJUST_DEFAULT |
TOKEN_ADJUST_SESSIONID
);
[DllImport("kernel32.dll", SetLastError = true, ExactSpelling = true)]
public static extern IntPtr GetCurrentProcess();
[DllImport("kernel32.dll", SetLastError = true, ExactSpelling = true)]
public static extern IntPtr GetCurrentThread();
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Auto)]
public static extern bool LookupPrivilegeValue(string lpSystemName, string lpName, out LUID lpLuid);
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Auto)]
public static extern bool AdjustTokenPrivileges(IntPtr TokenHandle, bool DisableAllPrivileges, ref TOKEN_PRIVILEGES NewState, UInt32 BufferLengthInBytes, IntPtr PreviousStateNull, IntPtr ReturnLengthInBytesNull);
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Auto)]
public static extern bool OpenProcessToken(IntPtr ProcessHandle, UInt32 DesiredAccess, out IntPtr TokenHandle);
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Auto)]
public static extern bool OpenThreadToken(IntPtr ThreadHandle, UInt32 DesiredAccess, bool OpenAsSelf, out IntPtr TokenHandle);
[DllImport("ntdll.dll", EntryPoint = "RtlAdjustPrivilege")]
public static extern int RtlAdjustPrivilege(
UInt32 Privilege,
bool Enable,
bool CurrentThread,
ref bool Enabled
);
[DllImport("Kernel32.dll", SetLastError = true)]
public static extern bool CloseHandle(IntPtr handle);
//
//
private static LUID LookupPrivilege(string privilegeName)
{
LUID privilegeValue = new LUID();
bool res = LookupPrivilegeValue(null, privilegeName, out privilegeValue);
if (!res)
{
throw new Exception("Error: LookupPrivilegeValue()");
}
return privilegeValue;
}
//
//
public static void AdjustPrivilege(string privilegeName, bool enable)
{
IntPtr accessToken = IntPtr.Zero;
bool res = false;
try
{
LUID privilegeValue = LookupPrivilege(privilegeName);
res = OpenThreadToken(GetCurrentThread(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, false, out accessToken);
if (!res)
{
res = OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, out accessToken);
if (!res)
{
throw new Exception("Error: OpenProcessToken()");
}
}
TOKEN_PRIVILEGES tokenPrivileges = new TOKEN_PRIVILEGES();
tokenPrivileges.PrivilegeCount = 1;
tokenPrivileges.Luid = privilegeValue;
if (enable)
{
tokenPrivileges.Attributes = SE_PRIVILEGE_ENABLED;
}
else
{
tokenPrivileges.Attributes = 0;
}
res = AdjustTokenPrivileges(accessToken, false, ref tokenPrivileges, (uint)System.Runtime.InteropServices.Marshal.SizeOf(tokenPrivileges), IntPtr.Zero, IntPtr.Zero);
if (!res)
{
throw new Exception("Error: AdjustTokenPrivileges()");
}
}
finally
{
if (accessToken != IntPtr.Zero)
{
CloseHandle(accessToken);
accessToken = IntPtr.Zero;
}
}
}
}
}
'@
if ([object]::Equals(('SystemPrivilege.Win32API.Privileges' -as [type]), $null)) {
Add-Type -TypeDefinition $win32api
}
}
includeSystemPrivileges;
$privileges|%{[SystemPrivilege.Win32API.Privileges]::AdjustPrivilege($_, $true)}
# Validation
whoami /priv|?{$_ -match "SeBackupPrivilege|SeRestorePrivilege"}
}
addSystemPrivilege;
################################## Add System Backup Privileges ####################################
# This function increases the default windows 260 characters path length limit to 1024
Function remove260CharsPathLimit([string]$computerName=$env:computername){
$osVersion=[System.Environment]::OSVersion.Version
$dotNetVersion=(get-itemproperty 'REGISTRY::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full\' -Name "Version").Version
$windowsManagementFramework=$PSVersionTable.PSVersion
$osFeasible=$osVersion -ge [version]'10.0.1607'
$dotNetFeasible=$dotNetVersion -ge [version]'4.6.2'
$windowsFrameworkFeasible=$windowsManagementFramework -ge [version]'5.1'
if(!$osFeasible){
write-warning "OS Version $osVersion cannot be registry fixed to work with long file paths"
}
if(!$dotNetFeasible){
write-warning "Dot Net Version $dotNetVersion cannot be registry fixed to work with long file paths"
}
if(!$windowsFrameworkFeasible){
write-warning "Windows Management Framework $windowsManagementFramework cannot be registry fixed to work with long file paths"
}
if($osFeasible -and $dotNetFeasible -and $windowsFrameworkFeasible){
# Declare static variables
$registryHive="SYSTEM\CurrentControlSet\Control\FileSystem"
$keyName = 'LongPathsEnabled'
$value= 1
$remoteRegistry = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey("LocalMachine",$computerName)
$currentValue=$remoteRegistry.OpenSubKey("$registryHive\$keyName").GetValue($keyName)
if($currentValue -ne $value){
$remoteHive = $remoteRegistry.OpenSubKey($registryHive,$True)
$remoteHive.CreateSubKey($keyName)
$remoteKeyValue = $remoteRegistry.OpenSubKey("$registryHive\$keyName",$True)
$remoteKeyValue.SetValue($keyName,$value)
# Validation
$resultKey=$remoteRegistry.OpenSubKey("$registryHive\$keyName")
if($resultKey.GetValue($keyName) -eq $value){"\\$computerName\HKLM\$registryHive\$keyName has been added successfully."}
}else{
write-host "$computername already has $keyName set as $value"
}
return $true
}else{
return $false
}
}
remove260CharsPathLimit
################################## Commence File Copy Operations ####################################
function validatePaths{
param($inputData)
$thisClusterName=(get-wmiobject -class "MSCluster_Cluster" -namespace "root\mscluster" -ErrorAction SilentlyContinue|select -ExpandProperty Name|Out-String).Trim();
$objectLength=$inputData.length;
$castedArrayList=[System.Collections.ArrayList]$inputData;
# Remove row if any path doesn't resolve. Overcome limitations of Powershell's immutable array "fixed size" using this workaround
function removeItemFromArray{
param($arrayList,$removeAtIndex)
$arrayList.RemoveAt($removeAtIndex);
}
for ($i=0;$i -lt $objectLength; $i++){
#$from=$inputData[$i].From;
#$referenceCluster=$inputData[$i].Clustername;
$from=$castedArrayList[$i].From;
$referenceCluster=$castedArrayList[$i].Clustername;
if(!($thisClusterName -eq $referenceCluster)){
write-host "Removing $from from Array because of cluster mismatch...";
removeItemFromArray -arrayList $castedArrayList -removeAtIndex $i;
$objectLength--;
$i--;
}else{
if(!(test-path $from)){
write-host "Removing $from from Array because of unresolvable paths...";
removeItemFromArray -arrayList $castedArrayList -removeAtIndex $i;
$objectLength--;
$i--;
}
}
}
$inputData=[Array]$castedArrayList; #reverse the casting and reassign to original Array
$inputData=$inputData|Sort-Object -Property @{Expression = {$_.From}; Ascending = $true},To # PowerShell 2.0 backward compatible method
return $inputData
}
# function to obtain the version of a particular excecutable
function getExecutableVersion{
param(
[string]$computername=$env:computername,
[string]$executablename="robocopy.exe"
)
$localComputerName=$env:computername
$isLocal=if($computername -match "(localhost|127.0.0.1|$localComputerName)"){$true}else{$false}
if ($isLocal){
$getExecutable=get-command $executablename
if ($getExecutable.Count -ge 1){
$exeInfo=(get-item (get-command $executablename)[0].Definition).versionInfo;
}else{
$exeInfo=(get-item (get-command $executablename).Definition).versionInfo;
}
}else{
$exeInfo=invoke-command -computername $computername -scriptblock{
param($executable)
$getExecutable=get-command $executablename
if ($getExecutable.Count -ge 1){
$exeInfo=(get-item (get-command $executablename)[0].Definition).versionInfo;
}else{
$exeInfo=(get-item (get-command $executablename).Definition).versionInfo;
}
return $exeInfo;} -args $executablename
}
$exeVersion=$exeInfo.ProductVersion;
$exeLocation=$exeInfo.FileName;
$fileVersion=$exeInfo.FileVersion;
return "$executablename`: release $exeVersion $(if($exeVersion -ne $fileVersion){"| version '$fileVersion'"}) | location $exeLocation"
}
function generateErrorsLog{
$rawLog=Get-Content -Path $logFile;
$regexErrors="( : ERROR \()|( : WARNING : )"
$errorsFound="";
$rawLog | % {
$errorLine=$_ -match $regexErrors;
if ($errorLine){
$errorsFound+="$_`r`n"
}
}
if($errorsFound){
$errorsFound="`r`n===================================== ERRORS Section Begins ======================================`r`n"+$errorsFound+"===================================== ERRORS Section Ends ======================================`r`n"
Add-Content $logFile $errorsFound
}
}
function stopAnyService{
param(
[string]$serviceName,
[string]$computerName=$env:computername
)
function executeKillCommand($serviceName){
write-host "including prerequisites..."
.{
# Prerequisite commands
$chocoAvailable="get-command choco -ErrorAction SilentlyContinue";
$psexecAvailable="get-command psexec -ErrorAction SilentlyContinue";
$setAclAvailable="get-command setacl -ErrorAction SilentlyContinue";
if (!(Invoke-Expression $chocoAvailable)) {
write-host "Installing Choco...";
Set-ExecutionPolicy Bypass -Scope Process -Force;
iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));
}
if (!(Invoke-Expression $chocoAvailable)) {
write-host "Unable to install Chocolatey automation tool. Program now aborts.";
break;
}
if(!(Invoke-Expression $psexecAvailable)){
$pendingRebootTests = @(
@{
Name = 'RebootPending'
Test = { Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing' -Name 'RebootPending' -ErrorAction SilentlyContinue }
TestType = 'ValueExists'
}
@{
Name = 'RebootRequired'
Test = { Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update' -Name 'RebootRequired' -ErrorAction SilentlyContinue }
TestType = 'ValueExists'
}
@{
Name = 'PendingFileRenameOperations'
Test = { Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager' -Name 'PendingFileRenameOperations' -ErrorAction SilentlyContinue }
TestType = 'NonNullValue'
}
)
foreach ($test in $pendingRebootTests) {
$pendingReboot=Invoke-Command -ScriptBlock $test.Test
if($pendingReboot){
write-host "$env:computername currently has a pending reboot requirement. Aborting session...";
break;
}
}
try {
$status = ([wmiclass]"\\.\root\ccm\clientsdk:CCM_ClientUtilities").DetermineIfRebootPending()
if(($status -ne $null) -and $status.RebootPending){
write-host "$env:computername currently has a pending reboot requirement. Aborting session...";
break;
}
}catch{}
write-host "Installing PSExec...";
choco install sysinternals -y -force;
}
if (!(Invoke-Expression $psexecAvailable)) {
write-host "Unable to install psexec. Program now aborts.";
break;
}
if(!(Invoke-Expression $setAclAvailable)){
write-host "Installing setacl...";
choco install setacl -y -force;
}
if (!(Invoke-Expression $setAclAvailable)) {
write-host "Unable to install setACL. Program now aborts.";
break;
}
write-host "Done.";
}
function forceKillService ($service){
if ($service.Status -ne "Stopped"){
# Try to stop the service as normal - only modify permissions and retry upon encountering errors
try{
Stop-Service $serviceName -Force -ErrorAction Stop;
}
catch{
$serviceRunas=(Get-WMIObject win32_service |?{$_ -like "*$serviceName*"}).StartName;
<# $serviceRunas=(Get-CIMInstance win32_service |?{$_ -like "*$serviceName*"}).StartName;
The term 'Get-CIMInstance' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:16
+ Get-CIMInstance <<<< win32_service
+ CategoryInfo : ObjectNotFound: (Get-CIMInstance:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
#>
write-host "$serviceName seems to be owned by $serviceRunas. Now seizing permissions...";
# Grant permissions of service to the Administrators group
$nullOutput=PSExec -s -accepteula SetACL.exe -on $serviceName -ot srv -actn ace -ace 'n:Administrators;p:full' 2>&1; #redirect (>) 'stderr'(2) messages to 'stdout'(1); where 1 is a file descriptor (&), not a file
write-host "Process name $serviceName has been granted access to the Administrators group.";
# Retry stopping service
try{
Stop-Service $serviceName -Force;
write-host "$serviceName has been stopped successfully.";
return $true;
}
catch{
write-host $Error;
write-host "$serviceName has NOT been stopped successfully.";
return $false;
}
}
}else{
write-host "$serviceName is already stopped.";
return $true;
}
}
$matches=get-service|?{$_.DisplayName -like "*$serviceName*" -or $_.Name -like "*$serviceName*" -or $_.Servicename -like "*$serviceName*"};
#$matches=get-service|?{$_.DisplayName -like "*$serviceName*" -or $_.Name -like "*$serviceName*"};
if (!($matches)){
$message="$serviceName doesn't match anything on $env:computername";
write-host $message;
return $false;
}
if($matches.count -gt 1){
# This is a PowerShell 2.0 backward compatible technique to rebuild an object with a new column of Index values
$displayMatches=for ($i=0;$i -lt $matches.count;$i++){
$matches[$i]|Select-Object @{name='Index';e={$i}},Name,Displayname,Status
}
$displayMatches=$displayMatches|ft -AutoSize|Out-String
write-host "We have multiple matches for the $servicename`:`r`n-----------------------------------------------------------------$displayMatches";
$input=Read-Host "Please pick an index number from the above list"
write-host "Index value $input received.";
$matches=$matches[$input];
if (!($matches)){
write-host "Index value $input is invalid. No actions were taken.";
return $false;
}else{
$service=$matches;
forceKillService $service;
return $true;
}
}else{
$service=$matches;
forceKillService $service;
return $true;
}
}
function initPsSessionAsAdmin($computerName){
function checkAdminPrivileges{
param($credential)
$myWindowsID=[System.Security.Principal.WindowsIdentity]::GetCurrent()
$currentUser=$myWindowsID.Name
$myWindowsPrincipal=new-object System.Security.Principal.WindowsPrincipal($myWindowsID)
$adminRole=[System.Security.Principal.WindowsBuiltInRole]::Administrator;
if ($myWindowsPrincipal.IsInRole($adminRole))
{
write-host "$currentUser is an Administrator. Program will now initiate a new session with such account...";
$Host.UI.RawUI.BackgroundColor = "Black";
return $true;
}
else
{
return $false;
}
}
function getAdminCredentials{
$exitLoop=$false;
do {
$credential= get-credential;
if(checkAdminPrivileges -credential $credential){$exitLoop=$true};
sleep 1;
}while ($exitLoop -eq $false)
return $credential;
}
if (!(checkAdminPrivileges)){
$cred=getAdminCredentials
try{
$session=New-PSSession -Credential $cred -ComputerName $computerName;
}catch{
# Enable WinRM as try block command didn't succeed;
start-process powershell.exe -credential $cred -nonewwindow -ArgumentList "enable-psremoting -force";
refreshenv;
$session=New-PSSession -Credential $cred -ComputerName $computerName;
}
}else{
try{
$session=New-PSSession -ComputerName $computerName;
}catch{
# Enable WinRM as try block command didn't succeed;
start-process powershell.exe -credential $cred -nonewwindow -ArgumentList "winrm quickconfig -force;"
refreshenv;
$session=New-PSSession -ComputerName $computerName;
}
}
if ($session){
return $session;
}else{
write-host "Could not proceed due to errors in the process of initiating a new PSSession.";
break;
}
}
function invokeKillCommand{
param($service)
# Cleanup any lingering PS Sessions
get-pssession|remove-pssession;
if (!$adminPsSession -or $adminPsSession.State -eq "Closed"){$adminPsSession=initPsSessionAsAdmin -computerName $computerName;}
if(!(get-command psexec -ea SilentlyContinue)){
if (!(get-command choco -ea SilentlyContinue)) {
write-host "Installing Choco...";
Set-ExecutionPolicy Bypass -Scope Process -Force;
iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));
}
write-host "Installing PSExec...";
choco install sysinternals -y -force;
}
# Setting WinRM memory size to ensure success
#Correct format: winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="$ramMB"}'
#Alternaltive: psexec \\$computerName PowerShell Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 1024 #This one runs into permission issues
[int]$ramMB=(gwmi -Class win32_operatingsystem -computername $computerName -ea stop).TotalVisibleMemorySize/1024
if ($ramMB -ge 4090){$ramMB=4090}
$nullOutput=invoke-expression "psexec \\$computerName -s winrm.cmd set winrm/config/winrs '@{MaxMemoryPerShellMB=`"$ramMB`"}'" 2>&1;
invoke-command -Session $adminPsSession -ScriptBlock{
param($importedFunc,$x)
[ScriptBlock]::Create($importedFunc).invoke($x);
} -args ${function:executeKillCommand},$service
Remove-PSSession $adminPsSession;
}
function killService{
param($service)
$computerNameRegex='^(.*?)\.'
$local=$([void]($computerName -match $computerNameRegex);if($matches){$matches[1]}else{$computerName}) -like $env:computername;
if ($local){
executeKillCommand -service $service;
}else{
invokeKillCommand -service $service;
}
}
killService -service $serviceName
}
function checkDiskFreeSnapshotFeasibility{
param(
[string]$computerName,
[string]$localPath
)
<#
Excerpt from https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/ee692290(v=ws.10)?redirectedfrom=MSDN:
"For volumes less than 500 megabytes, the minimum is 50 megabytes of free space.
For volumes more than 500 megabytes, the minimum is 320 megabytes of free space.
It is recommended that least 1 gigabyte of free disk space on each volume if the volume size is more than 1 gigabyte."
#>
# Generate variables
if(!($computerName)){$thisNode="localhost"}else{$thisNode=$computerName}
$thisVolume=$localPath.SubString(0,2)
# Obtain disk information
$diskObject = Get-WmiObject Win32_LogicalDisk -ComputerName $thisNode -Filter "DeviceID='$thisVolume'"
$diskFree=[Math]::Round($diskObject.FreeSpace / 1MB)
$diskSize=[Math]::Round($diskObject.Size / 1MB)
switch ($diskSize){
{$diskSize -ge 1024} {if ($diskFree -gt 1024){$feasible=$True;}else{$feasible=$False;};;break;}
{$diskSize -ge 500} {if ($diskFree -gt 320){$feasible=$True;}else{$feasible=$False;};;break;}
{$diskSize -lt 500} {if ($diskFree -gt 50){$feasible=$True;}else{$feasible=$False;};break;}
}
return $feasible
}
function createShadow{
[cmdletbinding()]
param(
[string]$targetVolume="C:\",
[string]$snapshotAccessPoint='C:\vssSnapshot'
)
# Sanitation
if (!($targetVolume -like "*\")){$targetVolume+="\"}
$shadowMount=$snapshotAccessPoint;
if(Test-Path $shadowMount){(Get-Item $shadowMount).Delete()}
write-host "Initiating VSS snapshot..."
$shadowCopyClass=[WMICLASS]"root\cimv2:win32_shadowcopy"
$thisSnapshot = $shadowCopyClass.Create($targetVolume, "ClientAccessible")
$thisShadow = Get-WmiObject Win32_ShadowCopy | Where-Object { $_.ID -eq $thisSnapshot.ShadowID }
$thisShadowPath = $thisShadow.DeviceObject + "\"
if (!($thisShadowPath)){
$allShadows=vssadmin list shadows;
$matchedIdIndex=$allShadows.IndexOf($allShadows -match $thisSnapshot.ShadowID);
$matchedLine=$allShadows[$matchedIdIndex+2];
$thisShadowPath=$([void]($matchedLine -match "(\\\\.*)$");$matches[1])
}
# Creating symlink
$voidOutput=cd C:
$voidOutput=cmd /c mklink /d $shadowMount $thisShadowPath | Out-Null
write-host "Shadow of $targetVolume has been made and it's accessible at this local file system (LFS): $shadowMount."
# Validation
if(Test-Path $shadowMount){
$snapshotId=$thisShadow.ID;
write-host "Snapshot $snapshotId has been created.";
return $snapshotId;
}else{
write-host "Failed to create client accessible VSS Snapshot.";
return $false;
}
}
function deleteShadow{
[cmdletbinding()]
param(
[string]$targetVolume="C:\",
[string]$snapShotId,
[string]$snapshotAccessPoint='C:\vssSnapshot'
)
# Obtain the newest snapshot ID if it is not specified
if(!($snapShotId)){
$snapShotId=vssadmin list shadows /for=$targetVolume|`
%{if($_ -like "*Shadow Copy ID:*"){$_}}|`
select-object -last 1|`
%{[void]($_ -match "Shadow Copy ID: (.*)$"); $matches[1]}
}
# Remove a single snapshot
write-host "Removing snapshot Id $snapShotId..."
#$removeSnapshotCommand="cmd.exe /c vssadmin delete shadows /Shadow=$snapShotId /quiet"
#$voidOutput=cmd.exe /c vssadmin delete shadows /Shadow=$lastSnapshotId /quiet
# This is the workaround to the annoyance of antivirus software terminating sessions upon invoking the snapshot removal procedure
$newSession=New-PSSession
Invoke-Command -Session $newSession -ScriptBlock{param($snapShotId);vssadmin delete shadows /Shadow=$snapShotId /quiet} -args $snapShotId
$newSession|Remove-PSSession
<#
invoke-expression 'cmd /c start powershell -Command {
param($snapShotId);
$command="cmd.exe /c vssadmin delete shadows /Shadow=$snapShotId /quiet";
write-host $command;
invoke-expression $command;
pause; } -Args $snapShotId'
#>
<# This is the workaround to the annoyance of antivirus software terminating sessions upon invoking the snapshot removal procedure
$newSession=New-PSSession
Invoke-Command -Session $newSession -ScriptBlock{param($snapShotId);vssadmin delete shadows /Shadow=$snapShotId /quiet} -args $snapShotId
$newSession|Remove-PSSession
#>
# Remove symlink
write-host "Removing symlink $snapshotAccessPoint..."
if(test-path $snapshotAccessPoint){
try{
(Get-Item $snapshotAccessPoint).Delete()
}catch{
[IO.Directory]::Delete($snapshotAccessPoint); # This is the preferred method to remove reparse points
}}
# Remove all Snapshots
#Get-WmiObject Win32_ShadowCopy | % {$_.delete()}
#vssadmin delete shadows /For=$targetVolume /Quiet
# Validation
#vssadmin list shadows /for=$targetVolume
$validateLastSnapshot=vssadmin list shadows /for=$targetVolume|`
%{if($_ -like "*Shadow Copy ID:*"){$_}}|`
select-object -last 1|`
%{[void]($_ -match "{(.*)}$"); $matches[1]}
$validateLastSnapshotId="{$validateLastSnapshot}"
if(!($validateLastSnapshotId -eq $snapShotId)){
write-host "Last snapshot Id is now $validateLastSnapshotId"
return $true
}else{
write-host "$snapShotId still exists. There was an error in its removal";
return $false
}
}
# This function depends on another external function 'addTls'; thus, it is not meant to be invoked remotely at this time
function triggerFastCopy{
param(
[string]$from,
[string]$to,
[string]$logFile,
[bool]$verify
)
# Autogen variables
$copyEngine="fastcopy";
# Start timer
#$fastCopyTimer=[System.Diagnostics.Stopwatch]::StartNew();
# Input sanitation note:
# a. The lack of proceeeding '\' of a source path means that the parent folder will be copied over to the destination.
# b. To only copy contents inside a source container, it is necessary to include that '\' character.
# c. Since this is not obvious to many users, input transformation is necessary to standardize copying behavior to be consistent with the description (b)
if($from[$from.length-1] -ne "\"){$from=$from+"\"}
$excludes="/exclude='\`$RECYCLE.BIN\;\System Volume Information\'"
$logDirectory=Split-Path $logFile -Parent
if (!(test-path $logDirectory -ea SilentlyContinue)){
New-Item -path $logDirectory -type directory -Force;
}
# Ensure that the copying engine is available in the environmental paths
if (!(Get-Command choco.exe -ErrorAction SilentlyContinue)) {
addTLS12;
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
}
if (!(get-command $copyEngine -ea SilentlyContinue)){choco install $copyEngine -y;}
if (!(get-command $copyEngine -ea SilentlyContinue)){
write-host "System does not have $copyEngine installed. Program now aborts.";
break;
}
$freeMemoryMb=(gwmi -Class win32_operatingsystem -ea stop).FreePhysicalMemory/1024
$percentDesired=0.8
$bufferSize=if([int]($freeMemoryMb*$percentDesired)){[int]($freeMemoryMb*0.8)}else{
$freeRamLegacy=(systeminfo | Select-String 'Available Physical Memory:').ToString().Split(':')[1].Trim();
$freeRamLegacy -match "(\d+),(\d+)";
$bufferSize=[int]([float]($matches[1]+$matches[2])*$percentDesired);
}
if ($verify){
$fastCopyCommand="fastcopy /cmd=sync /no_ui /acl /stream /verify /error_stop=FALSE /skip_empty_dir=FALSE /speed=full /bufsize=$bufferSize $excludes /filelog='$logFile' '$from' /to='$to'";
}else{
$fastCopyCommand="fastcopy /cmd=sync /no_ui /acl /stream /error_stop=FALSE /skip_empty_dir=FALSE /speed=full /bufsize=$bufferSize $excludes /filelog='$logFile' '$from' /to='$to'";
}
Invoke-Expression $fastCopyCommand|out-null;
#$runtime=$fastCopyTimer.Elapsed.Totalhours;
#$fastCopyTimer.Stop();
#return $runtime;
}
function transformLocalSourcePath{
param($originalSourcePath,$accessPath)
$splicedSourcePath=$originalSourcePath.substring(2)
$transformedPath=$accessPath+$splicedSourcePath;
return $transformedPath;
}
# Enable TLS 1.2
function enableTls12{
$windowsVersionNumber=[System.Environment]::OSVersion.Version.Major;
$windowsName=(Get-CimInstance -ClassName Win32_OperatingSystem).Caption
$dotNetSecurityProtocols=[enum]::GetNames([Net.SecurityProtocolType]);
write-host "These are the available protocols on this $windowsName machine:`r`n$dotNetSecurityProtocols";
if ($windowsVersionNumber -lt 10){
$tls12Available=[System.Net.ServicePointManager]::SecurityProtocol -match "(Tls12|3072)";
if(!($tls12Available)){
write-host "This legacy Windows version does not have TLS 1.2 capability. Now adding that feature...";
$hive="REGISTRY::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols"
$key="TLS 1.2"
$key1="Client"; $key1Dword1="DisabledByDefault"; $key1Dword1Value=0; $key1Dword2="Enabled"; $key1Dword2Value=1;
$key2="Server"; $key2Dword1="DisabledByDefault"; $key2Dword1Value=0; $key2Dword2="Enabled"; $key2Dword2Value=1;
if(!(Test-Path "$hive\$key")){New-Item -Path "$hive\$key" -Force | Out-Null}
if(!(Test-Path "$hive\$key\$key1")){New-Item -Path "$hive\$key\$key1" -Force | Out-Null}
if(!(Test-Path "$hive\$key\$key2")){New-Item -Path "$hive\$key\$key2" -Force | Out-Null}
New-ItemProperty -Path "$hive\$key\$key1" -Name $key1Dword1 -Value $key1Dword1Value -PropertyType DWORD -Force | Out-Null
New-ItemProperty -Path "$hive\$key\$key1" -Name $key1Dword2 -Value $key1Dword2Value -PropertyType DWORD -Force | Out-Null
New-ItemProperty -Path "$hive\$key\$key2" -Name $key2Dword1 -Value $key2Dword1Value -PropertyType DWORD -Force | Out-Null
New-ItemProperty -Path "$hive\$key\$key2" -Name $key2Dword2 -Value $key2Dword2Value -PropertyType DWORD -Force | Out-Null
# PowerShell 2.0: explicit casting of TLS12 and apply it to session
$castTLS12 = [Enum]::ToObject([System.Net.SecurityProtocolType], 3072);
[System.Net.ServicePointManager]::SecurityProtocol += $castTLS12;
}else{
write-host "This legacy Windows version already has TLS 1.2 capability added.";
}
}else{ #Windows 10 / 2016 Server and above
$tls12Available=[Enum]::GetNames([Net.SecurityProtocolType]) -contains 'Tls12';
if(!($tls12Available)){
write-host "Setting TLS 1.1 & 1.2 as the default security protocols.";
$hiveInternet32bit="REGISTRY::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings"
#$hiveInternet64bit="REGISTRY::HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Internet Settings"
$hiveInternetKey="WinHttp"
$hiveInternetDword="DefaultSecureProtocols"
$hiveInternetDwordValue="0x00000A00" #Tls 1.1 & 1.2
New-ItemProperty -Path "$hiveInternet32bit\$hiveInternetKey" -Name $hiveInternetDword -Value $hiveInternetDwordValue -PropertyType DWORD -Force | Out-Null
#New-ItemProperty -Path "$hiveInternet64bit\$hiveInternetKey" -Name $hiveInternetDword -Value $hiveInternetDwordValue -PropertyType DWORD -Force | Out-Null
}
# Enable TLS1.2 as Default
$tls12InUse=[System.Net.ServicePointManager]::SecurityProtocol.HasFlag([Net.SecurityProtocolType]::Tls12);
if (!($tls12InUse)){
write-host "Setting TLS 1.2 as the default security protocol for the dotNet Framework...";
$hive32bit="REGISTRY::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319";$key32bit="SchUseStrongCrypto";$key32bitValue=1
$hive64bit="REGISTRY::HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319";$key64bit="SchUseStrongCrypto";$key64bitValue=1
New-ItemProperty -Path $hive32bit -Name $key32bit -Value $key32bitValue -PropertyType DWORD -Force | Out-Null
New-ItemProperty -Path $hive64bit -Name $key64bit -Value $key64bitValue -PropertyType DWORD -Force | Out-Null
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
}
}
# Set default protocol to TLS 1.2 and bypass certificate trust issues
$classTrustAllCerts = @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(ServicePoint srvPoint, X509Certificate certificate,WebRequest request, int certificateProblem) {
return true;
}
}
"@
Add-Type -TypeDefinition $classTrustAllCerts
$trustAllCertsPolicy=New-Object TrustAllCertsPolicy
[System.Net.ServicePointManager]::CertificatePolicy = $trustAllCertsPolicy
try{[System.Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12}catch{}
}
function includePrerequisites{
function installChoco{
enableTls12
if (!(Get-Command choco.exe -ErrorAction SilentlyContinue)) {
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
}
}
function installJacksum{
installChoco;
choco install jacksum -y;
}
function installFastCopy{
installChoco;
choco install fastcopy -y;
}
function installEmcopy{
$emcopyIsInstalled=(Get-Command emcopy.exe -ErrorAction SilentlyContinue) # Deterministic check on whether emcopy is already available on this system
if (!($emcopyIsInstalled)){
$tempDir="C:\Temp"
$extractionDir="C:\Windows"
$emCopySource = "https://blog.kimconnect.com/wp-content/uploads/2019/08/emcopy.zip"
$destinationFile = "$tempDir\emcopy.zip";
try{[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12}catch{}
New-Item -ItemType Directory -Force -Path $tempDir
New-Item -ItemType Directory -Force -Path $extractionDir
$webclient = New-Object System.Net.WebClient;
$WebClient.DownloadFile($emCopySource,$destinationFile);
expandZipfile $destinationFile -Destination $extractionDir
}else{
"EMCOPY is currently available in this system.`n"
}
}
if(!(get-command jacksum -ErrorAction SilentlyContinue)){installJacksum}
if(!(get-command fastcopy -ErrorAction SilentlyContinue)){installFastcopy}
if(!(Get-Command emcopy.exe -ea silentlycontinue)){installEmcopy}
$ready=(get-command jacksum -ErrorAction SilentlyContinue) -and (get-command fastcopy -ErrorAction SilentlyContinue) -and (Get-Command emcopy -ea silentlycontinue)
return $ready
}
# Included function, not yet being called by any routines
function Get-STFolderSize {
<#
.SYNOPSIS
Gets folder sizes using COM and by default with a fallback to robocopy.exe, with the
logging only option, which makes it not actually copy or move files, but just list them, and
the end summary result is parsed to extract the relevant data.
.DESCRIPTION
There is a -ComOnly parameter for using only COM, and a -RoboOnly parameter for using only
robocopy.exe with the logging only option.
The robocopy output also gives a count of files and folders, unlike the COM method output.
The default number of threads used by robocopy is 8, but I set it to 16 since this cut the
run time down to almost half in some cases during my testing. You can specify a number of
threads between 1-128 with the parameter -RoboThreadCount.
Both of these approaches are apparently much faster than .NET and Get-ChildItem in PowerShell.
The properties of the objects will be different based on which method is used, but
the "TotalBytes" property is always populated if the directory size was successfully
retrieved. Otherwise you should get a warning (and the sizes will be zero).
Online documentation: https://www.powershelladmin.com/wiki/Get_Folder_Size_with_PowerShell,_Blazingly_Fast.php
MIT license. https://opensource.org/license/MIT
Copyright (C) 2015-present, Joakim Borger Svendsen
All rights reserved.
Svendsen Tech.
.PARAMETER Path
Path or paths to measure size of.
.PARAMETER LiteralPath
Path or paths to measure size of, supporting wildcard characters
in the names, as with Get-ChildItem.
.PARAMETER Precision
Number of digits after decimal point in rounded numbers.
.PARAMETER RoboOnly
Do not use COM, only robocopy, for always getting full details.
.PARAMETER ComOnly
Never fall back to robocopy, only use COM.
.PARAMETER RoboThreadCount
Number of threads used when falling back to robocopy, or with -RoboOnly.
Default: 16 (gave the fastest results during my testing).
.PARAMETER ExcludeDirectory
Names and paths for directories to exclude, as supported by the RoboCopy.exe /XD syntax.
To guarantee its use, you need to use the parameter -RoboOnly. If COM is used, nothing
is filtered out.
.PARAMETER ExcludeFile
Names/paths/wildcards for files to exclude, as supported by the RoboCopy.exe /XF syntax.
To guarantee its use, you need to use the parameter -RoboOnly. If COM is used, nothing
is filtered out.
.EXAMPLE
Import-Module .\Get-FolderSize.psm1
PS C:\> 'C:\Windows', 'E:\temp' | Get-STFolderSize
.EXAMPLE
Get-STFolderSize -Path Z:\Database -Precision 2
.EXAMPLE
Get-STFolderSize -Path Z:\Users\*\AppData\*\Something -RoboOnly -RoboThreadCount 64
.EXAMPLE
Get-STFolderSize -Path "Z:\Database[0-9][0-9]" -RoboOnly
.EXAMPLE
Get-STFolderSize -LiteralPath 'A:\Full[HD]FloppyMovies' -ComOnly
Supports wildcard characters in the path name with -LiteralPath.
.EXAMPLE
PS C:\temp> dir -dir | Get-STFolderSize | select Path, TotalMBytes |
Sort-Object -Descending totalmbytes
Path TotalMBytes
---- -----------
C:\temp\PowerShellGetOld 1.2047
C:\temp\git 0.6562
C:\temp\Docker 0.3583
C:\temp\MergeCsv 0.1574
C:\temp\testdir 0.0476
C:\temp\DotNetVersionLister 0.0474
C:\temp\temp2 0.0420
C:\temp\WriteAscii 0.0328
C:\temp\tempbenchmark 0.0257
C:\temp\From PS Gallery Benchmark 0.0253
C:\temp\RemoveOldFiles 0.0238
C:\temp\modules 0.0234
C:\temp\STDockerPs 0.0216
C:\temp\RandomData 0.0205
C:\temp\GetSTFolderSize 0.0198
C:\temp\Benchmark 0.0151
.LINK
https://www.powershelladmin.com/wiki/Get_Folder_Size_with_PowerShell,_Blazingly_Fast
#>
[CmdletBinding(DefaultParameterSetName = "Path")]
param(
[Parameter(ParameterSetName = "Path",
Mandatory = $True,
ValueFromPipeline = $True,
ValueFromPipelineByPropertyName = $True,
Position = 0)]
[Alias('Name', 'FullName')]
[String[]] $Path,
[System.Int32] $Precision = 4,
[Switch] $RoboOnly,
[Switch] $ComOnly,
[Parameter(ParameterSetName = "LiteralPath",
Mandatory = $true,
Position = 0)] [String[]] $LiteralPath,
[ValidateRange(1, 128)] [Byte] $RoboThreadCount = 16,
[String[]] $ExcludeDirectory = @(),
[String[]] $ExcludeFile = @())
Begin {
$DefaultProperties = 'Path', 'TotalBytes', 'TotalMBytes', 'TotalGBytes', 'TotalTBytes', 'DirCount', 'FileCount',
'DirFailed', 'FileFailed', 'TimeElapsed', 'StartedTime', 'EndedTime'
if (($ExcludeDirectory.Count -gt 0 -or $ExcludeFile.Count -gt 0) -and -not $ComOnly) {
$DefaultProperties += @("CopiedDirCount", "CopiedFileCount", "CopiedBytes", "SkippedDirCount", "SkippedFileCount", "SkippedBytes")
}
if ($RoboOnly -and $ComOnly) {
Write-Error -Message "You can't use both -ComOnly and -RoboOnly. Default is COM with a fallback to robocopy." -ErrorAction Stop
}
if (-not $RoboOnly) {
$FSO = New-Object -ComObject Scripting.FileSystemObject -ErrorAction Stop
}
function Get-RoboFolderSizeInternal {
[CmdletBinding()]
param(
# Paths to report size, file count, dir count, etc. for.
[String[]] $Path,
[System.Int32] $Precision = 4)
begin {
if (-not (Get-Command -Name robocopy -ErrorAction SilentlyContinue)) {
Write-Warning -Message "Fallback to robocopy failed because robocopy.exe could not be found. Path '$p'. $([datetime]::Now)."
return
}
}
process {
foreach ($p in $Path) {
Write-Verbose -Message "Processing path '$p' with Get-RoboFolderSizeInternal. $([datetime]::Now)."
$RoboCopyArgs = @("/L","/S","/NJH","/BYTES","/FP","/NC","/NDL","/TS","/XJ","/R:0","/W:0","/MT:$RoboThreadCount")
if ($ExcludeDirectory.Count -gt 0) {
$RoboCopyArgs += @(@("/XD") + @($ExcludeDirectory))
}
if ($ExcludeFile.Count -gt 0) {
$RoboCopyArgs += @(@("/XF") + @($ExcludeFile))
}
[DateTime] $StartedTime = [DateTime]::Now
[String] $Summary = robocopy $p NULL $RoboCopyArgs | Select-Object -Last 8
[DateTime] $EndedTime = [DateTime]::Now
#[String] $DefaultIgnored = '(?:\s+[\-\d]+){3}'
[Regex] $HeaderRegex = '\s+Total\s*Copied\s+Skipped\s+Mismatch\s+FAILED\s+Extras'
[Regex] $DirLineRegex = "Dirs\s*:\s*(?<DirCount>\d+)(?<CopiedDirCount>\s+[\-\d]+)(?<SkippedDirCount>\s+[\-\d]+)(?:\s+[\-\d]+)\s+(?<DirFailed>\d+)\s+[\-\d]+"
[Regex] $FileLineRegex = "Files\s*:\s*(?<FileCount>\d+)(?<CopiedFileCount>\s+[\-\d]+)(?<SkippedFileCount>\s+[\-\d]+)(?:\s+[\-\d]+)\s+(?<FileFailed>\d+)\s+[\-\d]+"
[Regex] $BytesLineRegex = "Bytes\s*:\s*(?<ByteCount>\d+)(?<CopiedBytes>\s+[\-\d]+)(?<SkippedBytes>\s+[\-\d]+)(?:\s+[\-\d]+)\s+(?<BytesFailed>\d+)\s+[\-\d]+"
[Regex] $TimeLineRegex = "Times\s*:\s*.*"
[Regex] $EndedLineRegex = "Ended\s*:\s*(?<EndedTime>.+)"
if ($Summary -match "$HeaderRegex\s+$DirLineRegex\s+$FileLineRegex\s+$BytesLineRegex\s+$TimeLineRegex\s+$EndedLineRegex") {
New-Object PSObject -Property @{
Path = $p
TotalBytes = [Decimal] $Matches['ByteCount']
TotalMBytes = [Math]::Round(([Decimal] $Matches['ByteCount'] / 1MB), $Precision)
TotalGBytes = [Math]::Round(([Decimal] $Matches['ByteCount'] / 1GB), $Precision)
TotalTBytes = [Math]::Round(([Decimal] $Matches['ByteCount'] / 1TB), $Precision)
BytesFailed = [Decimal] $Matches['BytesFailed']
DirCount = [Decimal] $Matches['DirCount']
FileCount = [Decimal] $Matches['FileCount']
DirFailed = [Decimal] $Matches['DirFailed']
FileFailed = [Decimal] $Matches['FileFailed']
TimeElapsed = [Math]::Round([Decimal] ($EndedTime - $StartedTime).TotalSeconds, $Precision)
StartedTime = $StartedTime
EndedTime = $EndedTime
CopiedDirCount = [Decimal] $Matches['CopiedDirCount']
CopiedFileCount = [Decimal] $Matches['CopiedFileCount']
CopiedBytes = [Decimal] $Matches['CopiedBytes']
SkippedDirCount = [Decimal] $Matches['SkippedDirCount']
SkippedFileCount = [Decimal] $Matches['SkippedFileCount']
SkippedBytes = [Decimal] $Matches['SkippedBytes']
} | Select-Object -Property $DefaultProperties
}
else {
Write-Warning -Message "Path '$p' output from robocopy was not in an expected format."
}
}
}
}
}
Process {
if ($PSCmdlet.ParameterSetName -eq "Path") {
$Paths = @(Resolve-Path -Path $Path | Select-Object -ExpandProperty ProviderPath -ErrorAction SilentlyContinue)
}
else {
$Paths = @(Get-Item -LiteralPath $LiteralPath | Select-Object -ExpandProperty FullName -ErrorAction SilentlyContinue)
}
foreach ($p in $Paths) {
Write-Verbose -Message "Processing path '$p'. $([DateTime]::Now)."
if (-not (Test-Path -LiteralPath $p -PathType Container)) {
Write-Warning -Message "$p does not exist or is a file and not a directory. Skipping."
continue
}
# We know we can't have -ComOnly here if we have -RoboOnly.
if ($RoboOnly) {
Get-RoboFolderSizeInternal -Path $p -Precision $Precision
continue
}
$ErrorActionPreference = 'Stop'
try {
$StartFSOTime = [DateTime]::Now
$TotalBytes = $FSO.GetFolder($p).Size
$EndFSOTime = [DateTime]::Now
if ($null -eq $TotalBytes) {
if (-not $ComOnly) {
Get-RoboFolderSizeInternal -Path $p -Precision $Precision
continue
}
else {
Write-Warning -Message "Failed to retrieve folder size for path '$p': $($Error[0].Exception.Message)."
}
}
}
catch {
if ($_.Exception.Message -like '*PERMISSION*DENIED*') {
if (-not $ComOnly) {
Write-Verbose "Caught a permission denied. Trying robocopy."
Get-RoboFolderSizeInternal -Path $p -Precision $Precision
continue
}
else {
Write-Warning "Failed to process path '$p' due to a permission denied error: $($_.Exception.Message)"
}
}
Write-Warning -Message "Encountered an error while processing path '$p': $($_.Exception.Message)"
continue
}
$ErrorActionPreference = 'Continue'
New-Object PSObject -Property @{
Path = $p
TotalBytes = [Decimal] $TotalBytes
TotalMBytes = [Math]::Round(([Decimal] $TotalBytes / 1MB), $Precision)
TotalGBytes = [Math]::Round(([Decimal] $TotalBytes / 1GB), $Precision)
TotalTBytes = [Math]::Round(([Decimal] $TotalBytes / 1TB), $Precision)
BytesFailed = $null
DirCount = $null
FileCount = $null
DirFailed = $null
FileFailed = $null
TimeElapsed = [Math]::Round(([Decimal] ($EndFSOTime - $StartFSOTime).TotalSeconds), $Precision)
StartedTime = $StartFSOTime
EndedTime = $EndFSOTime
} | Select-Object -Property $DefaultProperties
}
}
End {
if (-not $RoboOnly) {
[Void] [System.Runtime.Interopservices.Marshal]::ReleaseComObject($FSO)
}
[System.GC]::Collect()
#[System.GC]::WaitForPendingFinalizers()
}
}
function installXxHash{
$xxHashIsInstalled=(Get-Command xxhsum.exe -ErrorAction SilentlyContinue);
if (!($xxHashIsInstalled)){
# Set default protocol to TLS 1.2 and bypass certificate trust issues
$classTrustAllCerts = @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(ServicePoint srvPoint, X509Certificate certificate,WebRequest request, int certificateProblem) {
return true;
}
}
"@
try{
Add-Type -TypeDefinition $classTrustAllCerts
$trustAllCertsPolicy=New-Object TrustAllCertsPolicy
[System.Net.ServicePointManager]::CertificatePolicy = $trustAllCertsPolicy
}catch{}
try{
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12;
}catch{
[System.Net.ServicePointManager]::SecurityProtocol = [Enum]::ToObject([System.Net.SecurityProtocolType], 3072);
}
$tempDir="C:\Temp";
$extractionDir="C:\Windows"
$source = "https://github.com/Cyan4973/xxHash/releases/download/v0.7.2/xxhsum_win64_0_7_2_sse2.zip";
$destinationFile = "$tempDir\xxhash.zip";
New-Item -ItemType Directory -Force -Path $tempDir;
New-Item -ItemType Directory -Force -Path $extractionDir;
$webclient = New-Object System.Net.WebClient;
$WebClient.DownloadFile($source,$destinationFile);
try{
Expand-Archive -LiteralPath $destinationFile -DestinationPath $extractionDir;
}catch{
$shell = New-Object -ComObject Shell.Application
$zipFile = $shell.NameSpace($destinationFile);
$destinationFolder = $shell.NameSpace($extractionDir);
$copyFlags = 0x00
$copyFlags += 0x04 # Hide progress dialogs
$copyFlags += 0x10 # Overwrite existing files
$destinationFolder.CopyHere($zipFile.Items(), $copyFlags)
}
# Validation
$xxHashIsInstalled=(Get-Command xxhsum.exe -ErrorAction SilentlyContinue);
if ($xxHashIsInstalled){
write-host "xxhsum.exe has been added into environmental path of $extractionDir successfully."
}else{
write-host "There was an error preventing xxhsum.exe from being installed onto $env:computername.";
break;
}
}else{
write-host "xxHash is already available in $env:computername.`n";
}
}
# This function is not used until after "cutover"
function getFileHashes{
param (
[Parameter(Mandatory = $true, Position = 0)]
[string] $directory,
[Parameter(Mandatory = $false, Position = 1)]
[array] $files,
[Parameter(Mandatory = $false, Position = 2)]
[string] $threads=8,
[Parameter(Mandatory = $false, Position = 3)]
[string] $logFile
)
# Begin processing
$timer=[System.Diagnostics.Stopwatch]::StartNew();
if(!($files)){
# Be advised that this method of reading all file contents is extremely slow;
$message="Now obtaining hash signature of every file inside directory $directory...`r`nPlease be patient as this may take a while.";
write-host $message;
Add-Content $logFile $message;
$files = Invoke-Expression -Command:"cmd.exe /C dir '$directory' /S /B /W /A:-D"; # Tested speed on an iSCSI volume: 629.36 files per second or 2,265,715 files per hour
$fileCount=$files.Count;
$fileListingTime=[math]::round($timer.Elapsed.TotalHours,2);
$fileListMessage+="A list of $fileCount files has been generated with duration of $fileListingTime hours.`r`n";
Write-Host $fileListMessage;
Add-Content $logFile $fileListMessage;
}else{
$fileCount=$files.Count;
}
$filesArray=@();
$message="$directory has $fileCount files. Now splitting hash generation into $threads threads...";
write-host $message;
Add-Content $logFile $message;
# Execute simultaneous processes for optimal efficiency
if($fileCount){
# Generate eight equal fragments and assign to variable $indexes
#[uint32]$fileCount=4294967295
[uint32]$startIndex=0;
[uint32]$endIndex=0;
[int]$fragments=$threads;
[uint32]$fragmentCount=[math]::Floor(($fileCount/$fragments))
$indexes=@();
for ($i=0;$i -lt $fragments; $i++){
if($i -eq $fragments-1){
$startIndex=[uint32]($fragmentCount*$i);
$endIndex=$fileCount-1;
#write-host "startIndex: $startIndex | endIndex: $endIndex";
}else{
$startIndex=[uint32]($fragmentCount*$i);
$endIndex=$startIndex+$fragmentCount-1;
#write-host "startIndex: $startIndex | endIndex: $endIndex";
}
$indexes+=,@($startIndex,$endIndex);
}
$jobIds=@()
foreach ($index in $indexes){
[uint32]$beginIndex=$index[0];
[uint32]$endIndex=$index[1];
$filesArrayFragment=$files[$beginIndex..$endIndex]
$job=Start-Job -ScriptBlock {
param($filesArrayFragment)
$resultFilesArray=@();
# Get hash of each file
for ($j=0;$j -lt $filesArrayFragment.count;$j++){
$file=$filesArrayFragment[$j];
$hash=$([void]((xxhsum -q $file ) -match "^(.*?\s)");$matches[1]);
$resultFilesArray+=[PSCustomObject]@{File=$file;Hash=$hash};
}
return $resultFilesArray;
} -ArgumentList (,$filesArrayFragment) # This is how to pass an Array variable into an Argument list
$jobIds+=,$job.Id;
}
$message="These Job IDs have been started: $($jobIds -join ',')`r`nDepending on the number of files to process, this will be as fun as watching the grass grow...";
write-host $message;
Add-Content $logFile $message;
wait-job -id $jobIds|out-null;
$filesArray=Receive-Job -Id $jobIds # -AutoRemoveJob function is not available on PowerShell 2.0;
remove-job -Id $jobIds|out-null;
<#
# Run a sample of 1% of the files to get a speed estimate
$sampleTimer=[System.Diagnostics.Stopwatch]::StartNew();
[int]$tenPercentSample=$fileCount/10
for ($i=0;$i -lt $tenPercentSample;$i++){
$file=$files[$i];
$hash=$([void]((xxhsum.exe $file) -match "^(.*?\s)");$matches[1]);
$filesArray+=[PSCustomObject]@{File=$file;Hash=$hash};
}
$timeElapsedHours=[math]::round($sampleTimer.Elapsed.TotalHours,2);
$sampleSpeed=[math]::round($tenPercentSample/$sampleTimer.Elapsed.TotalHours,2);
$sampleTimer.stop();
$message="=======================================$tenPercentSample files have been processed in $timeElapsedHours hours: speed $sampleSpeed files/hour=======================================`r`n"
write-host $message;
Add-Content $logFile $message;
# Finish the rest of the files
for ($j=$tenPercentSample;$j -lt $fileCount;$j++){
$file=$files[$j];
$hash=$([void]((xxhsum $file) -match "^(.*?\s)");$matches[1]);
$filesArray+=[PSCustomObject]@{File=$file;Hash=$hash};
}
#>
$processTime=[math]::round($timer.Elapsed.TotalHours,2);
$processMessage+="A list of $fileCount files has been generated with duration of $processTime hours.`r`n";
Write-Host $processMessage;
Add-Content $logFile $processMessage;
return $filesArray;
}else{
$message="There appears to be an error as the file count for $directory has returned a null result;";
write-host $message;
Add-Content $logFile $message;
return @();
}
}
# copyMismatchedFiles -sourcePath $source -destinationPath $destination -sourceFiles $fileDifferences;
function copyMismatchedFiles{
param(
$sourcePath,
$destinationPath,
$sourceFiles,
$logFile
)
foreach ($file in $sourceFiles){
$correspondingDestinationFile=$destinationPath+$file.Substring($source.Length);
<# This doesn't work with robocopy XP010
$sourceParent=split-path $file -parent;
$destinationParent=Split-Path $correspondingDestinationFile -parent;
$fileName=Split-path $file -Leaf;
$copyCommand="robocopy.exe '$sourceParent' '$destinationParent' '$fileName' /XO /njh /njs /ndl /nc /ns /np 2>&1";
#>
$leftTimeStamp=[datetime](Get-ItemProperty -Path $file -Name LastWriteTime -ErrorAction SilentlyContinue).lastwritetime;
$rightTimeStamp=try{[datetime](Get-ItemProperty -Path $correspondingDestinationFile -Name LastWriteTime -ErrorAction SilentlyContinue).lastwritetime}catch{}
if ($leftTimeStamp -gt $rightTimeStamp){
$copyCommand="copy '$file' '$correspondingDestinationFile'";
invoke-expression $copyCommand;
$message="Copied: $file => $correspondingDestinationFile"
write-host $message;
Add-Content $logFile $message;
}else{
$message="Source timestamp $leftTimeStamp is older than Destination timestamp $rightTimeStamp => Skipping $file"
write-host $message;
Add-Content $logFile $message;
}
}
}
function fileCompare{
param($source,$destination,$logFile)
# Arbitrary variable for program to decide whether it's practical to perform hashes on this batch
$fileCompareMax=200000
# Sanitize
if ($source[$source.Length] -eq "\"){$source=$source.Substring(0,$source.Length-1)}
# Begin processing
$timer=[System.Diagnostics.Stopwatch]::StartNew();
$initMessage="=======================================File Compare has started for $source => $destination $(get-date)=======================================`r`n";
$initMessage+="Now collecting a list of files to be processed. Depending on the size of this batch, this process may take a long time..`r`n";
Write-Host $initMessage;
Add-Content $logFile $initMessage;
if ($bypassFileCompare){
$fileCount=$fileCompareMax+1
}else{
# Fastest way to list all files in a directory
$files = Invoke-Expression -Command:"cmd.exe /C dir '$source' /S /B /W /A:-D"; # Tested speed on an iSCSI volume: 629.36 files per second or 2,265,715 files per hour
$fileCount=$files.Count;
$fileListingTime=[math]::round($timer.Elapsed.TotalHours,2)
$fileListMessage+="A list of $fileCount files has been generated with duration of $fileListingTime hours.`r`n";
Write-Host $fileListMessage;
Add-Content $logFile $fileListMessage;
}
$isFileComparePractical=$fileCount -le $fileCompareMax;
if ($isFileComparePractical){
<# Generating this variable is much slower than running "cmd.exe /C dir '$source' /S /B /W /A:-D"
# Generating a list of corresponding destination files
$destinationFiles=@();
foreach ($file in $files){
#$correspondingFile=$destination+$($file[$source.Length..$file.Length] -join ""); # Slower than the Substring method
$correspondingFile=$destination+$file.Substring($source.Length,$file.Length-$source.Length); # This method is 75% more performant
$destinationFiles+=,$correspondingFile;
}
#>
$sourceIsLocal=$source -match "^[A-Za-z]\:";
if ($sourceIsLocal){
$sourceServer=$env:computername;
}else{
$sourceServer=$([void]($destination -match "^\\\\(.*?)\\");$matches[1];);
$sourceWinRmIsAvailable=Test-WSMan -ComputerName $sourceServer -ErrorAction silentlycontinue
if (!($sourceWinRmIsAvailable)){$sourceServer=$env:computername}
}
$destinationIsLocal=$destination -match "^[A-Za-z]\:";
if ($destinationIsLocal){
$destinationServer=$env:computername;
}else{
$destinationServer=$([void]($destination -match "^\\\\(.*?)\\");$matches[1];);
$remoteWinRmIsAvailable=Test-WSMan -ComputerName $destinationServer -ErrorAction silentlycontinue
if (!($remoteWinRmIsAvailable)){$destinationServer=$env:computername}
}
if ($sourceIsLocal -and $destinationIsLocal){
$sourceThreads=3;
$destinationThreads=3;
}else{
if ($sourceIsLocal){
$sourceThreads=6;
$destinationThreads=8;
}
if ($destinationIsLocal){
$sourceThreads=8;
$destinationThreads=6;
}
if (!$sourceIsLocal -and !$destinationIsLocal){
$sourceThreads=8;
$destinationThreads=8;
}
}
stopAnyService -serviceName "MsMpSvc" -computerName $env:computername;
stopAnyService -serviceName "MsMpSvc" -computerName $destinationServer;
$job1=invoke-command -computername $sourceServer -ScriptBlock{
param($addSystemPrivilege,$installXxHash,$getFileHashes,$directory,[array]$files,$threads,$logFile)
[ScriptBlock]::Create($addSystemPrivilege).invoke()|out-null;
[ScriptBlock]::Create($installXxHash).invoke();
$executionTime=(measure-command{$fileArray=[ScriptBlock]::Create($getFileHashes).invoke($directory,$files,$threads,$logFile)}).TotalHours
return @($executionTime,$fileArray)
} -args ${function:addSystemPrivilege},${function:installXxHash},${function:getFileHashes},$source,$(,$files),$sourceThreads,$logFile -AsJob
$job2=invoke-command -computername $destinationServer -ScriptBlock{
param($addSystemPrivilege,$installXxHash,$getFileHashes,$directory,$threads,$logFile)
[ScriptBlock]::Create($addSystemPrivilege).invoke()|out-null;
[ScriptBlock]::Create($installXxHash).invoke();
$executionTime=(measure-command{$fileArray=[ScriptBlock]::Create($getFileHashes).invoke($directory,@(),$threads,$logFile)}).TotalHours
return @($executionTime,$fileArray)
} -args ${function:addSystemPrivilege},${function:installXxHash},${function:getFileHashes},$destination,$destinationThreads,$logFile -AsJob
#$job1=Start-Job $processPath -ArgumentList $source
#$job2=Start-Job $processPath -ArgumentList $destination
$job1ID=$job1.ID;
$job2ID=$job2.ID;
# Wait for both jobs to complete
Wait-Job -Id $job1ID,$job2ID
#While ((Get-Job -id $job1ID -State "Running") -OR (Get-Job -id $job2ID -State "Running")){Start-Sleep 10}
# Getting the information back from the jobs
$sourceArray=Receive-Job -Id $job1.id #| Select-Object -Property * -ExcludeProperty PSComputerName,RunspaceID
$destinationArray=Receive-Job -Id $job2.id #| Select-Object -Property * -ExcludeProperty PSComputerName,RunspaceID
remove-job -Id $job1.id,$job2.id;
#$sourceArray=Receive-Job $job1 #| Select-Object -Property * -ExcludeProperty PSComputerName,RunspaceID
#$destinationArray=Receive-Job $job2 #| Select-Object -Property * -ExcludeProperty PSComputerName,RunspaceID
#$fileDifferences=(Compare-Object -ReferenceObject $sourceArray -DifferenceObject $destinationArray -Property Hash -PassThru).File #This notation doesn't work on PS2.0
# This code is to be backward compatible with Windows 2008
$job1Duration=$sourceArray[0]
$sourceFiles=$sourceArray[1]|%{$_.File}
$sourceHashes=$sourceArray[1]|%{$_.Hash}
$job2Duration=$destinationArray[0]
$destinationFiles=$destinationArray[1]|%{$_.File}
$destinationHashes=$destinationArray[1]|%{$_.Hash}
# Clear variables to save memory
Clear-Variable -Name sourceArray;
Clear-Variable -Name destinationArray;
$sourceFileCount=$sourceFiles.Count;
$destinationFileCount=$destinationFiles.Count;
#$job1Duration=($job1.PSEndTime-$job1.PSBeginTime).TotalHours; # These were not available on Windows 2008
#$job2Duration=($job2.PSEndTime-$job2.PSBeginTime).TotalHours;
$sourceSpeed=[math]::round($sourceFileCount/$job1Duration,2);
$destinationSpeed=[math]::round($destinationFileCount/$job2Duration,2);
$message="Source: $sourceFileCount files processed in $([math]::round($job1Duration,2)) hours v.s. Destination: $destinationFileCount files processed in $([math]::round($job2Duration)) hours`r`n";
$message+="Source files hashing collection speed: $sourceSpeed files/hour vs Destination files hashing collection speed: $destinationSpeed files/hour`r`n";
write-host $message;
Add-Content $logFile $message;
$fileCompareTime=[math]::round($timer.Elapsed.TotalHours,2)-$fileListingTime
$fileCompareMessage+="Total file compare duration:$fileCompareTime hours`r`n";
Write-Host $fileCompareMessage;
Add-Content $logFile $fileCompareMessage;
# Comparing arrays in PowerShell 2.0 compatible coding
$fileDifferences=@();
if ($destinationFileCount){
for ($i=0;$i -lt $sourceFiles.count;$i++){
$sourceFile=$sourceFiles[$i]
$sourceHash=$sourceHashes[$i]
$x=$source.length
$commonDenominator=$sourceFile.SubString($x,$sourceFile.length-$x);
$translatedFileName=$destination+$commonDenominator;
#$matchedIndex=$destinationFiles.IndexOf($translatedFileName); # This doesn't work on legacy systems
$matchedIndex=[array]::indexof($destinationFiles,$translatedFileName) # load the .NET function as workaround for older Windows
if($matchedIndex -eq -1){
$fileDifferences+=,$sourceFile;
}else{
$hashMismatch=$sourceHash -ne $destinationHashes[$matchedIndex];
if ($hashMismatch){$fileDifferences+=,$sourceFile;}
}
}
}else{
$destinationFileCount=0;
if ($sourceFileCount){
$fileDifferences=$sourceFiles;
}else{
write-host "Source directory is empty. Program now aborts.";
break;
}
}
#>
if($fileDifferences){
$copyMessage="There were files that don't match their destination counterparts:`r`n";
$copyMessage+="Please wait while the program copies mismatched files to their destinations.";
write-host $copyMessage;
Add-Content $logFile $copyMessage;
copyMismatchedFiles -sourcePath $source -destinationPath $destination -sourceFiles $fileDifferences -logFile $logFile;
}else{
$copyMessage="All of $sourceFileCount source files have matching signatures at their destinations! A perfect 100% score!";
write-host $copyMessage;
Add-Content $logFile $copyMessage;
}
<#
$time=[math]::round($timer.Elapsed.TotalHours,2);
$timer.Stop();
$timer.Reset();
$closingMessage="`r`nRun Time: $time hours"
write-host $closingMessage;
Add-Content $logFile $closingMessage;
#>
$totalDuration=[math]::round($timer.Elapsed.TotalHours,2)
$fileMergeTime=$totalDuration-$fileCompareTime-$fileListingTime
$fileProcessingMessage+="Overall process durations: File Listing ($fileListingTime) + File Compare ($fileCompareTime) + File Merge ($fileMergeTime) = $totalDuration hours.`r`n";
Write-Host $fileProcessingMessage;
Add-Content $logFile $fileProcessingMessage;
}else{
$exceptionMessage+="Program has detected that it's infeasible to run fileCompare. FastCopy will be triggered instead.`r`n";
Write-Host $exceptionMessage;
Add-Content $logFile $exceptionMessage;
triggerFastCopy -from $source -to $destination -logFile $logFile -verify $true;
$totalDuration=[math]::round($timer.Elapsed.TotalHours,2)
$fastCopyTime=$totalDuration-$fileListingTime
$fastCopyMessage+="Overall process durations: File Listing ($fileListingTime) + Fast Copy ($fastCopyTime) = $totalDuration hours`r`n";
Write-Host $fastCopyMessage;
Add-Content $logFile $fastCopyMessage;
}
}
function fastSync{
param($sourceAndDestination,$logFile)
$fastTimer=[System.Diagnostics.Stopwatch]::StartNew();
$arrLength=$sourceAndDestination.Count;
$initMessage="=======================================Fast-Sync mode has STARTED on $env:computername $(get-date)=======================================`r`n";
Write-Host $initMessage;
Add-Content $logFile $initMessage;
for ($i=0;$i -lt $arrLength; $i++){
$copyFromDirectory=$sourceAndDestination[$i].From;
$copyToDirectory=$sourceAndDestination[$i].To;
$passNumber=$i+1;
# Performing Fast Sync without verifications
$initMessage="=======================================Pass $passNumber of $arrLength Normal-sync has initiated $(get-date)`: $copyFromDirectory => $copyToDirectory=======================================`r`n";
Write-Host $initMessage;
Add-Content $logFile $initMessage;
if($copyFromDirectory[$copyFromDirectory.length-1] -ne '\'){$copyFromDirectory+='\';}
triggerFastCopy -from $copyFromDirectory -to $copyToDirectory -logFile $logFile -verify $false | Out-Null
$fastSyncMessage="=======================================Pass $passNumber of $arrLength Normal-sync has completed $(get-date)`: $copyFromDirectory => $copyToDirectory=======================================`r`n"
write-host $fastSyncMessage;
Add-Content $logFile $fastSyncMessage;
}
$timeElapsedHours=[math]::round($fastTimer.Elapsed.TotalHours,2);
$closingMessage="=======================================Fast-Sync mode has FINISHED on $env:computername $(get-date) - duration $timeElapsedHours hours=======================================`r`n"
write-host $closingMessage;
Add-Content $logFile $closingMessage;
$logLocationMessage+="=======================================Please check the log at this location: $logFile=======================================`r`n";
write-host $logLocationMessage;
}
function normalSync{
param($sourceAndDestination,$logFile,$snapshotAccessPath,$bypassFileCompare,$antivirusDisabled)
$timer=[System.Diagnostics.Stopwatch]::StartNew();
$arrLength=$sourceAndDestination.Count;
$initMessage="=======================================Normal-Sync mode has STARTED on $env:computername $(get-date)=======================================`r`n";
Write-Host $initMessage;
Add-Content $logFile $initMessage;
for ($i=0;$i -lt $arrLength; $i++){
$copyFromDirectory=$sourceAndDestination[$i].From;
$copyToDirectory=$sourceAndDestination[$i].To;
$passNumber=$i+1;
$processDisplay="=======================================Pass $passNumber of $arrLength initiated $(get-date)`: $copyFromDirectory => $copyToDirectory=======================================`r`n";
write-host $processDisplay;
Add-Content $logFile $processDisplay;
# This sequence is assuming local paths
$sourceVolume=$copyFromDirectory.Substring(0,3);
if($antivirusDisabled){
# Determine whether to take snapshots
[char]$previousDriveLetter=if($i -gt 0){($sourceAndDestination[$i-1].From)[0]}else{$null}
[char]$currentDriveLetter=$copyFromDirectory[0]
[char]$nextDriveLetter=try{$($sourceAndDestination[$i+1].From)[0]}catch{$null}
[bool]$takeSnapshot=($currentDriveLetter[0] -ne '\' -and $currentDriveLetter -ne $previousDriveLetter);
[bool]$removeSnapshot=($currentDriveLetter[0] -ne '\' -and $currentDriveLetter -ne $nextDriveLetter);
write-host "normalSync -sourceAndDestination `$sourceAndDestination -logFile $logFile -snapshotAccessPath $snapshotAccessPath -bypassFileCompare $bypassFileCompare"
write-host "TakeSnapshot: $takeSnapshot`r`nRemoveSnapshot: $removeSnapshot`r`nPrevious: $previousDriveLetter`r`nCurrent: $currentDriveLetter`r`nNext: $nextDriveLetter"
#pause
$localSourcePathTransformed=if($takeSnapshot){
try{
$feasible=checkDiskFreeSnapshotFeasibility -localPath $sourceVolume
if($feasible){
write-host "$sourceVolume disk feasiblity for VSS Snapshots: passed`r`n"
$GLOBAL:lastSnapshotId=createShadow -targetVolume $sourceVolume -snapshotAccessPoint $snapshotAccessPath
transformLocalSourcePath -originalSourcePath $copyFromDirectory -accessPath $snapshotAccessPath
}else{
$message="Program will use normal copying methods, which will skip open files!"
write-host $message
Add-Content $logFile $message
$copyFromDirectory
}
}catch{
$message="Program will use normal copying methods, which will skip open files!"
write-host $message
Add-Content $logFile $message
$copyFromDirectory
}
}elseif(!$takeSnapshot -and $null -ne $lastSnapshotId){
transformLocalSourcePath -originalSourcePath $copyFromDirectory -accessPath $snapshotAccessPath
}else{
$message="Program will use normal copying methods, which will skip open files!"
write-host $message
Add-Content $logFile $message
$copyFromDirectory
}
$sourcePath=if($localSourcePathTransformed[$localSourcePathTransformed.length-1] -ne '\'){$localSourcePathTransformed+'\'}else{$localSourcePathTransformed}
# $fastCopycommand="triggerFastCopy -from '$sourcePath' -to '$copyToDirectory' -logFile '$logFile' -verify `$False"
# write-host $fastCopycommand
# $null=Invoke-Expression $fastCopyCommand
# Exclude recycle bins and preserve file owners
$emcopySwitches="/o /s /de /sd /sdd /purge /secforce /r:1 /w:2 /th 32 /c /log+:C:\$logFile"
$sourceParentFolder=split-path $sourcePath -Parent
$cmdlet="emcopy.exe '$sourcePath' '$copyToDirectory' $emcopySwitches /xd '$sourceParentFolderSystem Volume Information' '$sourceParentFolder$RECYCLE.BIN'"
Invoke-Expression $cmdlet
if(!$bypassFileCompare){fileCompare -source $localSourcePathTransformed -destination $copyToDirectory -logFile $logFile | Out-Null}
if ($removeSnapshot){
if($lastSnapshotId){
deleteShadow -targetVolume $sourceVolume -snapshotId $lastSnapshotId -snapshotAccessPoint $snapshotAccessPath
}else{
deleteShadow -targetVolume $sourceVolume -snapshotAccessPoint $snapshotAccessPath
}
}
$processDisplay="=======================================Pass $passNumber of $arrLength completed $(get-date)`: $copyFromDirectory => $copyToDirectory=======================================`r`n";
write-host $processDisplay;
Add-Content $logFile $processDisplay;
}else{
write-host "Antivirus is enabled. No VSS Snapshots shall be used."
# Exclude recycle bins and preserve file owners
$emcopySwitches="/o /s /de /sd /sdd /purge /secforce /r:1 /w:2 /th 32 /c /log+:C:\$logFile"
$sourceParentFolder=split-path $copyFromDirectory -Parent
$cmdlet="emcopy.exe '$copyFromDirectory' '$copyToDirectory' $emcopySwitches /xd '$sourceParentFolderSystem Volume Information' '$sourceParentFolder$RECYCLE.BIN'"
Invoke-Expression $cmdlet
}
}
$time=[math]::round($timer.Elapsed.TotalHours,2);
$timer.Stop();
$timer.Reset();
$closingMessage="=======================================Normal-Sync mode has FINISHED on $env:computername $(get-date) - duration $time hours=======================================`r`n"
write-host $closingMessage;
Add-Content $logFile $closingMessage;
$logLocationMessage+="=======================================Please check the log at this location: $logFile=======================================`r`n";
write-host $logLocationMessage
}
function finalSync{
param($sourceAndDestination,$logFile,$snapshotAccessPath,$bypassFileCompare,$antivirusDisabled)
function removeBlockSmbFirewallRule($logFile){
$blockSmbRuleMatch=(netsh advfirewall firewall show rule name=all dir=in) -match "BlockSMB";
if ($blockSmbRuleMatch){
$removeSmbBlock="netsh advfirewall firewall delete rule name='BlockSMB' 2>&1";
Invoke-Expression $removeSmbBlock;
}else{
$removeSmbBlock="The 'BlockSMB' firewall rule currently doesn't exist.";
}
return $removeSmbBlock
}
function blockSmbSessions($logFile){
# Disable File and Printer Sharing
$blockSmbMessage="Disabling SMB ingress traffic at the firewall...";
$null=netsh advfirewall set allprofiles state on 2>&1;
$blockSmbMessage+=netsh advfirewall firewall add rule name='BlockSMB' protocol=TCP dir=in localport=445 action=block;
Write-Host $blockSmbMessage;
Add-Content $logFile $blockSmbMessage;
# Disconnect all connected SMB Sessions
$disconnectSessionMessage="Disconnecting all SMB sessions to quiescient files...";
do {
$disconnectResult=net session /delete /y
$disconnectSessionMessage+=$disconnectResult
$null=ping -n 1 127.0.0.1 2>&1;
} while($disconnectResult.count -ne 2)
Write-Host $disconnectSessionMessage;
Add-Content $logFile $disconnectSessionMessage;
}
$timer=[System.Diagnostics.Stopwatch]::StartNew();
$initMessage="=======================================Final-Sync mode has STARTED on $env:computername $(get-date)=======================================`r`n";
Write-Host $initMessage;
Add-Content $logFile $initMessage;
# Note: these global variables were defined at the header of this script
Send-MailMessage -From $sendFrom -to $sendTo -Subject "FinalSync Started" -Body "Please check server $env:computername" -SmtpServer $smtpServer -DeliveryNotificationOption OnSuccess
# Capture prior firewall statuses
$firewallAllProfiles=netsh advfirewall show allprofiles |?{$_ -match "State"}
$firewallDomainState=$([void]($firewallAllProfiles[0] -match "(ON|OFF)");$matches[1];)
$firewallPrivateState=$([void]($firewallAllProfiles[1] -match "(ON|OFF)");$matches[1];)
$firewallPublicState=$([void]($firewallAllProfiles[2] -match "(ON|OFF)");$matches[1];)
# Performing sync
blockSmbSessions -logFile $logFile | out-null
normalSync $sourceAndDestination $logFile $snapshotAccessPath $bypassFileCompare $antivirusDisabled
# Revert firewall back to previous settings
$removeSmbBlockMessage="Now reverting 'BlockSMB' firewall rule`r`n"
removeBlockSmbFirewallRule -logFile $logFile | out-null;
$removeSmbBlockMessage+=netsh advfirewall set domainprofile state $firewallDomainState 2>&1;
$removeSmbBlockMessage+=netsh advfirewall set privateprofile state $firewallPrivateState 2>&1;
$removeSmbBlockMessage+=netsh advfirewall set publicprofile state $firewallPublicState 2>&1;
write-host $removeSmbBlockMessage;
Add-Content $logFile $removeSmbBlockMessage;
$time=[math]::round($timer.Elapsed.TotalHours,2)
$closingMessage="=======================================Final-Sync mode has FINISHED on $env:computername $(get-date) - duration $time hours=======================================`r`n"
write-host $closingMessage
Add-Content $logFile $closingMessage
$logLocationMessage+="=======================================Please check the log at this location: $logFile=======================================`r`n"
write-host $logLocationMessage
Send-MailMessage -From $sendFrom -to $sendTo -Subject "FinalSync Completed" -Body "Please check server $env:computername" -SmtpServer $smtpServer -DeliveryNotificationOption OnSuccess
}
function simpleRoboCopy{
param($sourceAndDestination,$logFile)
# Process the copying operations for Windows 2003, 2000, & NT4
$timer2=[System.Diagnostics.Stopwatch]::StartNew();
for ($i=0;$i -lt $sourceAndDestination.count; $i++){
$from=$sourceAndDestination[$i].From;
$to=$sourceAndDestination[$i].To;
$passNumber=$i+1;
$processDisplay="=======================================Pass $passNumber of $arrLength`: $from => $to=======================================`r`n";
Add-Content $logFile $processDisplay;
#$commandString="robocopy '$from' '$to' /TBD /FFT /NS /NC /NDL /S /E /COPY:DATS /R:0 /W:0 $log";
$commandString="robocopy '$from' '$to' /TBD /FFT /NS /NC /NDL /S /E /COPY:DATS /PURGE /MIR /B /NP /XO /R:0 /W:0 $log"
write-host $commandString;
#cmd /c pause | out-null;
Invoke-Expression $commandString|out-null
$iterationCompleteMessage="=======================================Pass $passNumber of $arrLength` COMPLETED $from => $to=======================================`r`n";
Add-Content $logFile $iterationCompleteMessage;
}
$totalHours=[math]::Round($timer2.Elapsed.TotalHours,2);
$summary="`r`n==============================Total Hours: $totalHours==============================`r"
Add-Content $logFile $summary;
}
function proceed{
if(!includePrerequisites){
write-warning "Program cannot proceed without certain dependencies"
exit
}
[Array]$arr=validatePaths -inputData $arr;
if(!($null -ne $arr)){
write-host "Program will not proceed without valid sources and destinations.";
exit;
}
# Header section
[decimal]$powerShellVersion="$($PSVersionTable.PSVersion.Major)`.$($PSVersionTable.PSVersion.Minor)";
$osAndMemory = gwmi -Class win32_operatingsystem -ea stop| `
Select-Object @{Name="os";Expression={$_.Caption}},`
@{Name="Memory";Expression={$_.TotalVisibleMemorySize / 1048576}},`
@{Name="FreeMemory";Expression={$_.FreePhysicalMemory / 1048576}},`
@{Name = "Utilization"; Expression = {"{0:N2} %" -f ((($_.TotalVisibleMemorySize - $_.FreePhysicalMemory)*100)/ $_.TotalVisibleMemorySize) }}
if($osAndMemory){
#$os=$osAndMemory.os;
$memory=$osAndMemory.Memory;
$freeMemory=$osAndMemory.FreeMemory;
$memoryUtilization=$osAndMemory.Utilization;
}
$osName=(Get-WmiObject -class Win32_OperatingSystem).Caption;
$windowsVersionNumber=[System.Environment]::OSVersion.Version.Major;
$arrLength=$arr.Length
$robocopyVersion=getExecutableVersion -executablename 'robocopy';
$fastCopyVersion=getExecutableVersion -executablename 'fastcopy';
$initInfo="=============================================Job Started: $dateStamp=============================================`r`n";
$initInfo+="Source server name: $ENV:COMPUTERNAME`n";
$initInfo+="$osName`r`nPowershell version: $powerShellVersion`r`n$robocopyVersion`r`n$fastCopyVersion`r`n"
$initInfo+="Memory: $memory GB | Free: $freeMemory GB | Utilization: $memoryUtilization`r`n`r`n"
$initInfo+="Now processing the following operations:`r`n";
$initInfo+=for ($i=0;$i -lt $arrLength; $i++){
# Pass output values to $initInfo
$from=$arr[$i].From;
$to=$arr[$i].To;
$passNumber=$i+1;
"$passNumber`: $from => $to`r`n";
# Overload array with additional properties
#[char]$previousDriveLetter=$(try{$($arr[$i-1].From)[0]}catch{'-1'});
#[char]$currentDriveLetter=$from[0];
#[char]$nextDriveLetter=$(try{$($arr[$i+1].From)[0]}catch{'1'});
#[bool]$GLOBAL:takeSnapshot=($currentDriveLetter -ne '\' -and $currentDriveLetter -ne $previousDriveLetter);
#[bool]$GLOBAL:removeSnapshot=($currentDriveLetter -ne '\' -and $currentDriveLetter -ne $nextDriveLetter);
}
Write-Host $initInfo;
if(!(test-path $logPath -ea SilentlyContinue)){New-Item -ItemType Directory -Force -Path $logPath | Out-Null}
Add-Content $logFile $initInfo;
# Stop any previous stuck processes by the name of fastcopy
get-process -name 'fastcopy' -ea Ignore|stop-process -Force -ErrorAction SilentlyContinue
write-host "Checking on whether Antivirus is running as it will be a problem for VSS Snapshot removals..."
$antivirusDisabled=.{
$x=createShadow
deleteShadow -targetVolume c:\ -snapShotId $x -snapshotAccessPoint $snapShotPath
}
if ($windowsVersionNumber -ge 6){
if ($syncMode -eq 'finalSync'){
finalSync -sourceAndDestination $arr -logFile $logFile -snapshotAccessPath $snapShotPath $bypassFileCompare $antivirusDisabled
}elseif($syncMode -eq 'fastSync'){
fastSync -sourceAndDestination $arr -logFile $logFile
}else{
normalSync -sourceAndDestination $arr -logFile $logFile -snapshotAccessPath $snapShotPath $bypassFileCompare $antivirusDisabled
}
}else{
simpleRoboCopy -sourceAndDestination $arr -logFile $logFile;
}
}
proceed;
}
$command="dataSync `$syncMode `$arr `$bypassFileCompare '$snapShotPath' '$logPath'"
invoke-expression $command
Categories: