Skip to content

posts🔗

Painless Synchronization of Azure Blob Storage with AWS S3

Synchronization

Moving data between two cloud providers can be painful, and require more provider scripting if doing api calls. For this, you can benefit from a tool that abstracts the calls into a seamless synchronization tool.

I've used RClone before when needing to deduplicate several terabytes of data in my own Google Drive, so I figured I'd see if it could help me sync up 25GB of json files from Azure to S3.

Very happy to report it worked perfectly, and with only a couple minutes of refamilarizing myself with the tool setup.

Install RClone

For windows users, it's as easy as leveraging Chocolatey and running

choco upgrade rclone -y

Setup Providers

Go through rclone config dialogue and setup your cloud provider. In my case, I setup Azure as a provider to connect to blob storage, and then AWS with s3.

{{< admonition type="info" title="Cloud to Cloud" >}} Providers that support cloud to cloud based calls without copying locally are provided in the section for Optional Features where you can view the operations that support calls {{< /admonition >}}

Initialize Sync

rclone copy azure:containername s3:bucketname/keyprefix --log-level ERROR --progress --dry-run

Wrap-up

Take a look at this if you need a simple way to grab some data from one provider and leverage in another and you might want to save yourself some time on learning provider specific api calls. I've found tools like this, Terraform, and others that help abstract the api calls can be a great resource as you can leverage one syntax to work with two completely different providers and eliminate a lot of effort in coding.

AWS SSM PowerShell Script Automation

SSM Overview

I've found that working with a large number of environments in AWS can provide some interesting challenges for performing various tasks, in a way that scale.

When you begin to have dozens to hundreds of servers that you might need to provide a quick fix, the last thing you want to do is RDP into each and perform some type of scripted action.

AWS SSM (Systems Manager) provides a tremendous amount of functionality to help manage systems. It can perform tasks from running a script, installing an application, and other mundane administrative oriented tasks, to more complex state management, AMI automation, and other tasks that might go beyond the boundaries of virtual machine management.

I'll probably be unpacking a few of these areas over the next few posts, since my world has been heavily focused on SSM usage in the last months, and leveraging it is a must for those working heavily with EC2.

PowerShell Execution

For the first simple example, AWS SSM provides documents that wrap up various scripted actions and accept parameters. This can be something like Joining a domain or running a shell script.

In my case, I've had the need to change a registry setting, restart a windows service, or set an environment variable across an environment. I additionally wanted to set the target of this run as a tag filter, instead of providing instanceid, since this environment is rebuilt often as part of development.

The commands to execute scripts have one flaw that I abhor. I hate escaping strings. This probably comes from my focused effort on mastering dynamic t-sql 💩, at which point I quickly tried to avoid using dynamic sql as much as possible as I realized it was not the end all solution I started to think it was when I just started learning it.

With PowerShell and AWS SSM things could get even messier. You'd have to pass in the command and hope all the json syntax and escaping didn't error things out.

The solution

Write PowerShell as natively designed, and then encode this scriptblock for passing as an encoded command. I've found for the majority of my adhoc work this provided a perfect solution to eliminate any concerns on having to escape my code, while still letting me write native PowerShell in my Vscode editor with full linting and syntax checks.

Authenticate

Import-Module AWSPowershell.NetCore, PSFramework #PSFramework is used for better config and logging. I include with any work i do
$ProfileName = 'taco'
$region = 'us-west-1'
Initialize-AWSDefaultConfiguration -ProfileName $ProfileName -region $region

Create Your Command

In this section, I've provided a way to reference an existing function so the remote instance can include this function in the local script execution rather than having to copy and paste it into your command block directly. DRY for the win.

#----------------------------------------------------------------------------#
#                  Include this function in remote command                   #
#----------------------------------------------------------------------------#
$FunctionGetAWSTags = Get-Content -Path 'C:\temp\Get-AWSTags.ps1' -Raw
$command = {
  Get-Service 'codedeployagent' | Restart-Service -Verbose
}

Now that you have a script block, you can work on encoding. This encoding will prevent you from needing to concern yourself with escaping quotes, and you were able to write your entire script in normal editor without issues in linting.

#----------------------------------------------------------------------------#
#                   encode command to avoid escape issues                    #
#----------------------------------------------------------------------------#
[string]$CommandString = [string]::Concat($FunctionGetAWSTags, "`n`n", $Command.ToString())
$bytes = [System.Text.Encoding]::Unicode.GetBytes($CommandString)
$encodedCommand = [Convert]::ToBase64String($bytes)
$decodedCommand = [System.Text.Encoding]::Unicode.GetString([System.Convert]::FromBase64String($encodedCommand));
Write-PSFMessage -Level Debug -Message "Decoded Command: $($DecodedCommand)"

In my local script, I'll also include this Wait-SSM command that's a quick way to wait for the results of the SSM job to finish and show status. This is because Send-SSMCommand is actually an asynchronous command and doesn't wait for completion, just the successful sending of the command.

function Wait-SSM
{
    param(
        [Amazon.SimpleSystemsManagement.Model.Command]$Result
    )
    end
    {
        $Status = (Get-SSMCommandInvocation -CommandId $Result.CommandId -Details $true | Select-Object -ExpandProperty CommandPlugins).Status.Value
        while ($status -ne 'Success')
        {
            $Status = (Get-SSMCommandInvocation -CommandId $Result.CommandId -Details $true | Select-Object -ExpandProperty CommandPlugins).Status.Value
            Start-Sleep -Seconds 5
        }
        Get-SSMCommandInvocation -CommandId $Result.CommandId -Details $true | Select-Object InstanceId, Status | Format-Table -Autosize -Wrap
    }
}

Send Command

Finally, we get to the meat 🍗 and potatos... or in my case I'd prefer the meat and tacos 🌮 of the matter.

Sending the command...

$Message = (Read-Host "Enter reason")
$sendSSMCommandSplat = @{
    Comment                                       = $Message
    DocumentName                                  = 'AWS-RunPowerShellScript'
    #InstanceIds                                  = $InstanceIds # 50 max limit
    Target                                        = @{Key="tag:env";Values=@("tacoland")}
    Parameter                                     = @{'commands' = "powershell.exe -nologo -noprofile -encodedcommand $encodedCommand"  }
    CloudWatchOutputConfig_CloudWatchLogGroupName  = 'ssm/manual/my-command'
    CloudWatchOutputConfig_CloudWatchOutputEnabled = $true
}
$result = Send-SSMCommand  @sendSSMCommandSplat
Wait-SSM -Result $result

Note that you can also pass in an instance list. To do this, I'd recommend first filtering down based on tags, then also filtering down to available to SSM for running the command to avoid running on instances that are not going to succed, such as instances that are off, or ssm is not running on.

To stream results from cloudwatch, try looking at my post: Post on Using Cw for Cloudwatch Log Stream In Terminal

cw tail -f --profile=my-profile --region=eu-west-1 'ssm/manual/my-command'

EC2 Filters

To simplify working with tags, I often use the ConvertTo-Ec2Filter function that was written by David Christian (@dchristian3188) and can be viewed on this blog post EC2 Tags and Filtering.

Function ConvertTo-EC2Filter
{
    [CmdletBinding()]
    Param(
        [Parameter(
            ValueFromPipeline,
            ValueFromPipelineByPropertyName)]
        [HashTable]
        $Filter
    )
    Begin
    {
        $ec2Filter = @()
    }
    Process
    {
        $ec2Filter = Foreach ($key in $Filter.Keys)
        {
            @{
                name   = $key
                values = $Filter[$key]
            }
        }
    }
    End
    {
        $ec2Filter
    }
}
$searchFor = @{
    'tag:toppings'   = 'saucesAndMoreSauces'
    'tag:env'        = 'tacoland'
}


$ssmInstanceinfo        = Get-SSMInstanceInformation
$ec2Filter              = ConvertTo-EC2Filter -Filter $searchFor
$Instances              = @(Get-EC2Instance -Filter $ec2Filter).Instances
[string[]]$InstanceIds  = ($Instances | Where-Object { $_.State.Name -eq 'running' -and $_.InstanceId -in $ssmInstanceinfo.InstanceId } | Select-Object InstanceId -Unique).InstanceId

wrap-up

Hopefully this will get you going with Send-SSMCommand in a way that helps give you a simple way to issue commands across any number of EC2 instances. For me, it's saved a lot of manual console work to run commands against tagged environments, allowing me to more rapidly apply a fix or chocolatey package, or any number of needs in the context of testing, without all the overhead of doing per instances, or use the dreaded RDP 💩 connection.

If you find something unclear or worth more explanation, I'm always up for editing and refactoring this post. 🎉

Bump nuspec file version with powershell

Bump Nuspec Version

Bumping the version of the nuspec file requires a little tweaking and I got some help from the slack powershell community to ensure I handled the xml parsing correctly. This was the result. If you are running a chocolatey package build or equivalent nuspec build via an agent and want a way to ensure the latest build updates the build version incrementally this should help.

This snippet should help give you a way to bump a nuspec file version programmatically.

I modified the logic to support -WhatIf since I'm a fan of being able to run stuff like this without actually breaking things first.

Use Case

Leveraging aws system manager sessions can help with aws development, by eliminating the need to RDP for work that can be done via a PowerShell session. In addition, it can help bypass the need to use SSH tunneling, remote Windows management, or RDP hops into the final destination.

This leverages IAM Credentials, allowing consistent security management in alignment with other IAM policies, instead of having to manage another security setting like remote management would require, potentially reducing the security explore footprint.

  • Quickly access an instance that normally would require an additional hop, and then evaluate
  • Restart remote service without having to hop into it (or issue SSM prebuilt command docs)
  • Interact in other ways that are more adhoc in nature, and don't have prebuilt SSM Documents ready to go.

Browser

This is a great option for leveraging AWS Systems Manager web console. When you select start a session you'll be presented with the tagged instances by name that you can quickly select and start a remote session with.

Select Instances to Start Session Against

Start Session

Once you've started the session you'll enter into a remote prompt.

Interactive PowerShell Prompt on Remote Instance

Local Interactive Setup

I use Cmder for my main terminal, with all other terminals normally running in Visual Studio Code. If you open a Powershell session using the powershell plugin you can write your PowerShell in the editor, and the interactively run it in Visual Studio Code using the predefined F8 key.

Install on Windows

Ensure your AWS Credentials are setup, and use the session manager plugin after installation by running:

Start Session

aws ssm start-session --target MyInstanceId

Limitations

Refresh rate is slow. Input for large script blocks from Visual Studio Code is also really slow. This means that putting a local function in scope by running F8 against it and then wanting to run this function interactively can take a while.

The best use case I see is for adhoc administrative or investigative work. If larger scripts are required, then having a script setup to install module or copy from s3 would be a much more performance solution, as it wouldn't require large amounts of console text streaming.

2019-05-31T19:00:42.000+00:00

I'd read about Terraform 0.12 and thought it was much further out, so moved on with regret from evaluating the massive number of improvements. Just found out it was released, and choco upgrade terraform -y provided me with a delightful 0.12 upgrade. If you haven't explored it yet, go do it!

Things like loops, no longer having to reference any variable with string interpolation, and more promises to make this a big productivity improvement for those enjoying Terraform.

If you aren't using any Infrastructure-As-Code approach right now, you'll find it initially a little confusing, but get past that and you'll wonder how you ever lived without it.

2019-05-17T15:57:32-05:00

Really enjoying my experience with Terraform from the last month. If you have any resources in the cloud you have to deploy and you are having to do them manually, you should take a look. It's very easy to get going with the basics and the time it can save as you build up Terrachops (patent pending) can be tremendous.

2019-05-17T15:59:50-05:00

Kids learn so quick. It's amazing how fast my son has picked up #origami. He often has to wait for his slow-poke dad.

origami with son

2019-05-16T11:15:28-05:00

Downloading a Visual Studio Code vsix extension and then installing manually in Azure Data Studio works for some extensions! For instance, downloading Simple Alignment from the marketplace, and then running in Azure Data Studio successfully installed the utility.

You can also many of your keyboard settings straight from vscode into Azure Data Studio so you don't have to remap all those keys. The beauty of json configuration files 😄

2019-04-25 00:39:35 -0500

Finally upgraded my graphics card and hardrive. EVGA RTX 2080 and a Samsung Evo 970 NVMe 2TB SSD breathed new life into my PC. Running 100hz 3440x1440 ultrawide on a 10 year old AMD 6950 was just not doing the trick.

2019-04-19 22:57:00 -0500

Working smart: how great performers do less work

This book so far is a good read. I like the concept of the "feedback loop", and doing small iterative improvements with a targeted narrow focus to improve each day. It's very much in alignment with Agile concepts. It's kinda like delivery of small measurable bits of value for your own improvement. This contrasts our typical promise to ourselves of radical transformation or resolutions that never get realized. Clarifying these concepts is really helpful to help one be proactive instead of reactive about personal growth.