Syncing Files Using S5Cmd
Syncing Files Using S5Cmd🔗
Older versions of PowerShell (4.0) and the older AWSTools don't have the required ability to sync down a folder from a key prefix. This also performs much more quickly for a quick sync of files like backups to a local directory for initiating restores against.
In testing down by the tool creator, they showed this could saturate a network for an EC2 with full download speed and be up to 40x faster than using CLI with the benefit of being a small self-contained Go binary as well.
Once the download of the tooling is complete you can run a copy from s3 down to a local directory by doing something similar to this, assuming you have rights to the bucket. If you don't have rights, you'll want to set environment variables for the access key and secret key such as:
Import-Module aws.tools.common, aws.tools.SecurityToken Set-AWSCredential -ProfileName 'MyProfileName' -Scope Global $cred = Get-STSSessionToken -DurationInSeconds ([timespan]::FromHours(8).TotalSeconds) @" `$ENV:AWS_ACCESS_KEY_ID = '$($cred.AccessKeyId)' `$ENV:AWS_SECRET_ACCESS_KEY = '$($cred.SecretAccessKey)' `$ENV:AWS_SESSION_TOKEN = '$($cred.SessionToken)' "@
You can copy that string into your remote session to get the access tokens recognized by the s5cmd tool and allow you to grab files from another AWS account S3 bucket.
NOTE: To sync a full "directory" in s3, you need to leave the asteriks at the end of the key as demonstrated.
Install S5Cmd For Windows🔗
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 $ToolsDir = 'C:\tools' $ProgressPreference = 'SilentlyContinue' $OutZip = Join-Path $ToolsDir 's5cmd.zip' Invoke-WebRequest -Uri 'https://github.com/peak/s5cmd/releases/download/v1.2.1/s5cmd_1.2.1_Windows-64bit.zip' -UseBasicParsing -OutFile $OutZip # Not available on 4.0 Expand-Archive $OutZip -DestinationPath $ToolsDir Add-Type -assembly 'system.io.compression.filesystem' [io.compression.zipfile]::ExtractToDirectory($OutZip, $ToolsDir) $s5cmd = Join-Path $ToolsDir 's5cmd.exe' &$s5cmd version
Windows Sync A Directory From To Local🔗
$ErrorActionPreference = 'Stop' $BucketName = '' $KeyPrefix = 'mykeyprefix/anothersubkey/*' $Directory = "C:\temp\adhoc-s3-sync-$(Get-Date -Format 'yyyy-MM-dd')\$KeyPrefix" New-Item $Directory -ItemType Directory -Force -ErrorAction SilentlyContinue $s5cmd = Join-Path $ToolsDir 's5cmd.exe' Write-Host "This is what is going to run: `n&$s5cmd cp `"s3://$bucketname/$KeyPrefix`" $Directory" Read-Host 'Enter to continue if this makes sense, or cancel (ctrl+c)' &$s5cmd cp "s3://$bucketname/$KeyPrefix" $Directory