Made Of Things

As this is the first post on my blog, please feel free to read something about myself on the About page.

The challenge

While working with Windows Azure Pack (WAPack) and Service Management Automation (SMA) for the last 18 months, the biggest drawback for me: Version Control of Runbooks that are stored in SMA. In short: It’s not there. Yes, we have the “Draft” and “Publish” status, but that is really limited. Yes, we can export with PowerShell and store the Runbooks offline (i.e. Using SMART), it does a great job, but limited as it does not track the actual changes line-by-line inside your Runbook.

Beside TFS, there are some other great (free or paid) version control repositories available. Git being one of them. Either hosted on GitHub.com, build yourself on premise/build yourself in the Azure Cloud or get GitLab from the VM Depot (ready to run on Azure). Assumption is that you have one of these in place for your organization. When you have TFS already running as your version control repository, look here.

So how does Git work? That is probably too detailed to explain here, and me being not the best person to ask. Assuming your have your repository running in a few clicks, I found the following documentation a valuable resource to start reading: http://git-scm.com/doc

At least you need a Git Client. If you have PowerShell 5.0 (WMF5) you can use OneGet to get it from Chocolatey.org, or from the internal NuGet Repository in your organization. Or use the old fashioned download, get the Git Client from here: http://git-scm.com/download/win

Play around, explore, start with the following commands on your Windows command line: “git clone <url>”, “git init”, “git add .”, “git commit -m done”, “git push” and “git pull”.

And those are the commands we want to Automate in a SMA Runbook, to export all our valuable SMA Runbooks to a proper version control repository and keep track of all changes line-by-line. What I have done until now is one-way, assuming SMA is your “golden source” always pushing changes to Git.

Prerequisites

I assume that certain prerequisites are already in place:

• SMA Runbook workers on level UR4 with Windows Server 2012R2 and PowerShell 4.0
• The binaries of the Git Client copied to “C:\Git” on all your SMA Runbook workers. With “C:\Git\bin\git.exe” being the path to the executable, no need to install, just copy the installed “Git” directory to the SMA Runbook workers from a valid source (like your workstation, running PowerShell 5.0 that was getting it from Chocolately.org).
• A Git repository filled with at least 1 file (best practice: Place a file named “README.md” with the first line of text telling something about this repository)
• The URL for your Git repository, like “git@fqdnname.domain.com:USERNAME/REPOSITORY.git”
• The Public/Private keypair files (“.\id_rsa” and “.\id_rsa.pub”) without password, that is already working with your Git repository of choice via SSH. We will explain how to secure those keys with SMA.
• The “.\known_hosts” file which references to your Git repository being trustworthy.
• The firewall should be open from you SMA Runbook workers towards the Git repository.

 Preparing SMA

We also assume certain other things are in place before we can run our Runbook:

• A “Connection Asset” to SMA of the type “ServiceManagementAutomation” should exist:
  • Here is how to create one of the type: http://technet.microsoft.com/en-us/library/dn457809.aspx
  • Required to create it are: The account’s username and password, the SMA URL and SMA Port-number
  • The account used should have Administrative rights for SMA part of WAPack as is explained here: http://msdn.microsoft.com/en-us/library/dn688367.aspx
• A “Variable Asset” of the type Complex which is Encrypted:
  • How that works is explained here: https://mssecbyben.wordpress.com/2014/12/19/use-sma-complex-type-asset-to-store-product-keys-in-encrypted-form/
  • What we will put inside is the content of the files described in the prerequisite:
    • “.\id_rsa” and “.\id_rsa.pub” and “.\known_hosts” (these files are normally located in the folder “C:\Users\USERNAME\.ssh”)
  • We need to run a small script to create this Variable, detailed in the next section

 Creating a SMA Variable for the Key-pair files

One of the jobs in preparing SMA is to create a variable of the type Complex (Encrypted) with  content the following 3 files: Public Key, Private Key, Known Hosts. The script below does create this SMA Variable.

$WebServiceEndpoint="https://sma.domain.com"
$complexvarname="GitKey-Export-SMARunbooks"

$pvtfilename=".\id_rsa"
$pubfilename=".\id_rsa.pub"
$knownhostsfilename=".\known_hosts"

[string]$pvtmultiline=$null
[string]$pubmultiline=$null
[string]$knownhostsmultiline=$null
Get-Content $pvtfilename|%{$pvtmultiline+=($_+"`n")}
Get-Content $pubfilename|%{$pubmultiline+=($_+"`n")}
Get-Content $knownhostsfilename|%{$knownhostsmultiline+=($_+"`n")}

$keyscomplextype=@{
  "PrivateKey"=($pvtmultiline|ConvertTo-Json)
  "PublicKey"=($pubmultiline|ConvertTo-Json)
  "KnownHosts"=($knownhostsmultiline|ConvertTo-Json)
}

# Remove -Encrypted switch when Testing outside SMA
Set-SmaVariable -Name $complexvarname -Value $keyscomplextype -WebServiceEndpoint $WebServiceEndpoint -Encrypt

 

The actual Runbook

Note: The entire script can be found at the end of this post

I will not be discussing the Runbook in detail here. Basically is comes down to one big inlinescript after some rudimentary Variables as preperation.

Inside the inlinescript, you can distinguish the following sections:

• Preparing Windows (User) Environment variables
• Exporting the Keypair files (imported above)
• Starting the first git.exe commands to setup the Git environment: Username, Email and Default push mode
• Now switch to using sh.exe instead of git.exe (git is still used as part of the -c switch)

Reason: sh.exe automatically reads and uses the Keypair files correctly without having to setup additional command-line switches or Environment variables.

• Pulling your initial Clone from the Git repository with “–depth 1” as only need to control and push back the latest version and not years of history
• Removing all Runbooks from the $gitrunbooksubdir (“.\Runbooks”) directory  locally, that were just cloned/copied there using the “git clone <uri> –depth 1” command
• Newly Export all Published SMA Runbooks to the same, just cleared directory $gitrunbooksubdir
• Letting Git itself sort out the differences between the Removed files and newly Exported files:
  •  Using the commands “git add . –all”, “git commit -m <timestamp>” and “git push”
• Finally, cleaning up all temporary Files, Directories, Keys and Environment variables

The result

As an end result you will see the Runbook in your Git repository. Some quick screenshot below:

An example change of a Runbook editted in SMA. Showing below how you will find this back in Git:

Another tip

Schedule this Runbook, once a day, twice a day, hourly and never worry about losing track of changes in your Runbooks again.

The entire script (for download) to be used as SMA Runbook:

Note: Click on Expand, to Expand this section and see the code.

workflow Export-RunbooksToGit
{
    param (
        [string]$GitKey="GitKey-Export-SMARunbooks",
        [string]$GitSSHProjectURL="git@github.com:USERNAME/export-smarunbooks.git",
        [string]$GitProjectName="export-smarunbooks",
        [string]$GitRunbookSubdirName="Runbooks",
        [string]$GitWin32Path="C:\Git\bin",
        [string]$SmaConnectionName="SmaConnection"
    )
        
    # === BEGIN-SECTION FOR PREPARING VARIABLE RELATED TO SMARUNBOOK ENVIRONMENT ===
    $SMAConnection=Get-AutomationConnection -Name $SmaConnectionName

    $SMAUserName=inlinescript{($Using:SMAConnection).Get_Item("UserName")}
    $SMAUserPassword=inlinescript{($Using:SMAConnection).Get_Item("UserPassword")}
    $SMASecureUserPassword=ConvertTo-SecureString -AsPlainText -String $SMAUserPassword -Force
    $SMAWebServicePort=inlinescript{($Using:SMAConnection).Get_Item("WebServicePort")}
    $SMAWebServiceHostnameWithProtocol=inlinescript{($Using:SMAConnection).Get_Item("WebServiceHostnameWithProtocol")}
    $SMACredential=New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $SMAUserName,$SMASecureUserPassword        
    # === END-SECTION FOR PREPARING VARIABLE RELATED TO SMARUNBOOK ENVIRONMENT ===

    # Reading Public/Private keys from SMA Variable Assets
    $GitKeyComplex=Get-AutomationVariable -Name $GitKey    

    inlinescript {
        $VerbosePreference=[System.Management.Automation.ActionPreference]$Using:VerbosePreference
        $DebugPreference=[System.Management.Automation.ActionPreference]$Using:DebugPreference

        $workdir=[System.IO.Path]::GetTempPath()+[System.IO.Path]::GetRandomFileName()        

        $gitprojectdir=$workdir+"\"+$Using:GitProjectName
        $gittestdir=$gitprojectdir+"\.git\"
        $gitrunbooksubdir=$gitprojectdir+"\"+$Using:GitRunbookSubdirName

        $pvtsinglelinefromsma=$Using:GitDeployKeyComplex.PrivateKey|ConvertFrom-Json
        $pubsinglelinefromsma=$Using:GitDeployKeyComplex.PublicKey|ConvertFrom-Json
        $knownhostssinglelinefromsma=$Using:GitDeployKeyComplex.KnownHosts|ConvertFrom-Json
        $sshdir=$env:USERPROFILE+"\.ssh"
        $pvtfilename=$sshdir+"\id_rsa"
        $pubfilename=$sshdir+"\id_rsa.pub"
        $knownhostsfilename=$sshdir+"\known_hosts"

        # Create WORK directory
        if (-not (Test-Path $workdir)) {$tmpnewitem=New-Item -Force -ItemType Directory -Path $workdir}

        # Write SSH Security Keys required for Git connection
        if (-not (Test-Path $sshdir)) {$tmpnewitem=New-Item -Force -ItemType Directory -Path $sshdir}
        [IO.File]::WriteAllText($pvtfilename,$pvtsinglelinefromsma)
        [IO.File]::WriteAllText($pubfilename,$pubsinglelinefromsma)
        [IO.File]::WriteAllText($knownhostsfilename,$knownhostssinglelinefromsma)

        # $Using:GitWin32Bin add to environment 'path' (appending to 'current user')
        $ExistingUserpath=[System.Environment]::GetEnvironmentVariable("PATH","User")
        $UserpathElements=@([System.Environment]::GetEnvironmentVariable("PATH","User") -split ";")
        $UserpathElements+=$Using:GitWin32Path
        $NewUserPath=$UserpathElements -join ";"
        [System.Environment]::SetEnvironmentVariable("PATH",$NewUserPath,"User")

        # USERPROFILE and HOME environment varables are not available inside Workflow/inlinescript/Start-Process-combo
        [System.Environment]::SetEnvironmentVariable("HOME",$env:USERPROFILE,"User")
        [System.Environment]::SetEnvironmentVariable("USERPROFILE",$env:USERPROFILE,"User")

        #Start-Process -FilePath $env:ComSpec -Wait -UseNewEnvironment
        $OutputFile=[System.IO.Path]::GetTempFileName()
        $ErrorFile=[System.IO.Path]::GetTempFileName()

        "=-=-=configname"|Write-Debug
        $confignameargument="config --global user.name `"SMARBW "+$env:COMPUTERNAME+"`""
        Start-Process -FilePath ($Using:GitWin32Path+"\git.exe") -Wait -UseNewEnvironment -LoadUserProfile -RedirectStandardOutput $OutputFile  -RedirectStandardError $ErrorFile `
                -ArgumentList $confignameargument -WorkingDirectory $workdir
        Get-Content $OutputFile|where {$_ -ne "" -and $_ -notlike "Welcome to Git*" -and $_ -notlike "*git help*" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $OutputFile) {Remove-Item -Force -Path $OutputFile}
        Get-Content $ErrorFile|where {$_ -ne "" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $ErrorFile) {Remove-Item -Force -Path $ErrorFile}

        "=-=-=configemail"|Write-Debug
        $configemailargument="config --global user.email `""+$env:COMPUTERNAME+"@domain.com`""
        Start-Process -FilePath ($Using:GitWin32Path+"\git.exe") -Wait -UseNewEnvironment -LoadUserProfile -RedirectStandardOutput $OutputFile  -RedirectStandardError $ErrorFile `
                -ArgumentList $configemailargument -WorkingDirectory $workdir
        Get-Content $OutputFile|where {$_ -ne "" -and $_ -notlike "Welcome to Git*" -and $_ -notlike "*git help*" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $OutputFile) {Remove-Item -Force -Path $OutputFile}
        Get-Content $ErrorFile|where {$_ -ne "" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $ErrorFile) {Remove-Item -Force -Path $ErrorFile}

        "=-=-=configpushsimple"|Write-Debug
        Start-Process -FilePath ($Using:GitWin32Path+"\git.exe") -Wait -UseNewEnvironment -LoadUserProfile -RedirectStandardOutput $OutputFile  -RedirectStandardError $ErrorFile `
                -ArgumentList "config --global push.default simple" -WorkingDirectory $workdir
        Get-Content $OutputFile|where {$_ -ne "" -and $_ -notlike "Welcome to Git*" -and $_ -notlike "*git help*" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $OutputFile) {Remove-Item -Force -Path $OutputFile}
        Get-Content $ErrorFile|where {$_ -ne "" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $ErrorFile) {Remove-Item -Force -Path $ErrorFile}

        "=-=-=clone"|Write-Debug
        Start-Process -FilePath ($Using:GitWin32Path+"\sh.exe") -Wait `
                -RedirectStandardOutput $OutputFile  -RedirectStandardError $ErrorFile `
                -ArgumentList ("--login -c '/bin/git clone "+$Using:GitSSHProjectURL+" --depth 1'") -WorkingDirectory $workdir
        Get-Content $OutputFile|where {$_ -ne "" -and $_ -notlike "Welcome to Git*" -and $_ -notlike "*git help*" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $OutputFile) {Remove-Item -Force -Path $OutputFile}
        Get-Content $ErrorFile|where {$_ -ne "" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $ErrorFile) {Remove-Item -Force -Path $ErrorFile}

        # TEST SECTION - If the "git clone" command failed, the directory does not exist
        if (-not (Test-Path $gittestdir)) {
            "Error - This directory should exist after the git.exe-clone command, directory does not exist: "+$gittestdir|Write-Output

            # Cleanup (identical to Cleanup in error/exit section below)
            if (Test-Path $workdir) {Remove-Item -Force -Recurse -Path $workdir}
            if (Test-Path $sshdir) {Remove-Item -Force -Recurse -Path $sshdir}
            if (Test-Path ($Env:USERPROFILE+"\.gitconfig")) {
                Remove-Item -Force -Recurse -Path ($Env:USERPROFILE+"\.gitconfig")}
            if (Test-Path ($Env:USERPROFILE+"\.bash_history"))  {
                Remove-Item -Force -Recurse -Path ($Env:USERPROFILE+"\.bash_history")}
            if (Test-Path $OutputFile) {Remove-Item -Force -Path $OutputFile}
            if (Test-Path $ErrorFile) {Remove-Item -Force -Path $ErrorFile}
            [System.Environment]::SetEnvironmentVariable("PATH",$ExistingUserpath,"User")
            [System.Environment]::SetEnvironmentVariable("HOME",$null,"User")
            [System.Environment]::SetEnvironmentVariable("USERPROFILE",$null,"User")

            "Error - This directory should exist after the git.exe-clone command, directory does not exist: "+$gittestdir|Write-Error
        }

        # Although cloned, we need a Subdirectory for exporting the Runbooks to
        # But first, we want changes, also deletions of Runbooks, so let's remove everything :)
        if (Test-Path $gitrunbooksubdir) {Remove-Item -Force -Recurse -Path $gitrunbooksubdir}
        if (-not (Test-Path $gitrunbooksubdir)) {$tmpnewitem=New-Item -Force -ItemType Directory -Path $gitrunbooksubdir}

        # Exporting the Published Runbooks
        $runbooks=Get-SmaRunbook -WebServiceEndpoint $Using:SMAWebServiceHostnameWithProtocol -Port $Using:SMAWebServicePort -Credential $Using:SMACredential
        foreach ($runbook in $runbooks) {
                 Write-Debug "RunbookName/ID:"
                 $runbook.RunbookName|Write-Debug
                 $runbook.RunbookID|Write-Debug
                 $runbookdefinition=Get-SmaRunbookDefinition `
                    -Id $runbook.RunbookID `
                    -Type Published `
                    -WebServiceEndpoint $Using:SMAWebServiceHostnameWithProtocol `
                    -Port $Using:SMAWebServicePort `
                    -AuthenticationType "Windows" `
                    -Credential $Using:SMACredential
                 $runbookdefinition.Content|Set-Content -Path ($gitrunbooksubdir+"\"+$runbook.RunbookName+".ps1") -Force
        }

        # === BEGIN RUNBOOK OUTPUT ===
        # Pre-output for RunBook of what will be processed by the git.exe commands        
        $runbooks|select RunbookName,LastModifiedTime,LastModifiedBy|Sort-Object -Property RunbookName|ft -a|Out-File -FilePath $OutputFile -Force
        Get-Content $OutputFile
        "---"|Write-Output
        if (Test-Path $gitprojectdir) {
            Get-ChildItem -File $gitprojectdir|select Length,LastWriteTime,FullName|Sort-Object -Property FullName|ft -a|Out-File -FilePath $OutputFile -Force
            Get-Content $OutputFile
            "---"|Write-Output
        }

        if (Test-Path $gitrunbooksubdir) {
            Get-ChildItem -File $gitrunbooksubdir|select Length,LastWriteTime,Name|Sort-Object -Property Name|ft -a|Out-File -FilePath $OutputFile -Force
            Get-Content $OutputFile
            "---"|Write-Output
        }
        # === END RUNBOOK OUTPUT ===

        "=-=-=adddot"|Write-Debug
        Start-Process -FilePath ($Using:GitWin32Path+"\sh.exe") -Wait `
                -RedirectStandardOutput $OutputFile  -RedirectStandardError $ErrorFile `
                -ArgumentList ("--login -c '/bin/git add --all .'") -WorkingDirectory $gitprojectdir
        Get-Content $OutputFile|where {$_ -ne "" -and $_ -notlike "Welcome to Git*" -and $_ -notlike "*git help*" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $OutputFile) {Remove-Item -Force -Path $OutputFile}
        Get-Content $ErrorFile|where {$_ -ne "" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $ErrorFile) {Remove-Item -Force -Path $ErrorFile}

        "=-=-=commit"|Write-Debug
        $commitcomment=("RunBook-"+(Get-Date -Format o).Split("+")[0])
        Start-Process -FilePath ($Using:GitWin32Path+"\sh.exe") -Wait `
                -RedirectStandardOutput $OutputFile  -RedirectStandardError $ErrorFile `
                -ArgumentList ("--login -c '/bin/git commit -v -m "+$commitcomment+"'") -WorkingDirectory $gitprojectdir
        $keepcommitinformation=Get-Content $OutputFile
        Get-Content $OutputFile|where {$_ -ne "" -and $_ -notlike "Welcome to Git*" -and $_ -notlike "*git help*" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $OutputFile) {Remove-Item -Force -Path $OutputFile}
        Get-Content $ErrorFile|where {$_ -ne "" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $ErrorFile) {Remove-Item -Force -Path $ErrorFile}

        "=-=-=push"|Write-Debug
        Start-Process -FilePath ($Using:GitWin32Path+"\sh.exe") -Wait `
                -RedirectStandardOutput $OutputFile  -RedirectStandardError $ErrorFile `
                -ArgumentList "--login -c '/bin/git push'" -WorkingDirectory $gitprojectdir
        $keeppushinformation=Get-Content $OutputFile
        Get-Content $OutputFile|where {$_ -ne "" -and $_ -notlike "Welcome to Git*" -and $_ -notlike "*git help*" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $OutputFile) {Remove-Item -Force -Path $OutputFile}
        Get-Content $ErrorFile|where {$_ -ne "" -and $_ -notlike "#*"}|Write-Debug;if (Test-Path $ErrorFile) {Remove-Item -Force -Path $ErrorFile}

        # === BEGIN RUNBOOK OUTPUT 2nd ===
        # The output for RunBook of what "git commit" and "git push" actually did
        $keepcommitinformation|where {$_ -ne "" -and $_ -notlike "Welcome to Git*" -and $_ -notlike "*git help*" -and $_ -notlike "#*"}
        $keeppushinformation|where {$_ -ne "" -and $_ -notlike "Welcome to Git*" -and $_ -notlike "*git help*" -and $_ -notlike "#*"}
        # === END RUNBOOK OUTPUT 2nd ===

        # Cleanup (identical to Cleanup in error/exit section above)
        if (Test-Path $workdir) {Remove-Item -Force -Recurse -Path $workdir}
        if (Test-Path $sshdir) {Remove-Item -Force -Recurse -Path $sshdir}
        if (Test-Path ($Env:USERPROFILE+"\.gitconfig")) {
            Remove-Item -Force -Recurse -Path ($Env:USERPROFILE+"\.gitconfig")}
        if (Test-Path ($Env:USERPROFILE+"\.bash_history"))  {
            Remove-Item -Force -Recurse -Path ($Env:USERPROFILE+"\.bash_history")}
        if (Test-Path $OutputFile) {Remove-Item -Force -Path $OutputFile}
        if (Test-Path $ErrorFile) {Remove-Item -Force -Path $ErrorFile}
        [System.Environment]::SetEnvironmentVariable("PATH",$ExistingUserpath,"User")
        [System.Environment]::SetEnvironmentVariable("HOME",$null,"User")
        [System.Environment]::SetEnvironmentVariable("USERPROFILE",$null,"User")
    }
}