Skip to content



Kept getting an error Incorrect syntax near the keyword 'read' when running the some updated PowerShell 7 in lambda with the dotnet3.1 sdk. Was troubleshooting loading types thinking I was having path issues.

Turns out one of these things is not like the other. ๐Ÿคฃ

set nocount on
set transaction isolation read uncommitted
set nocount on
set transaction isolation level read uncommitted

I think this explains why "error during "read". Maybe I should have run in my Azure Data Studio session before trying serverless ๐Ÿ˜€


If you want to simplify keeping up with github releases instead of using email, try the GitHub slack app. Simple create a releases channel and then run these two steps:

  1. /github subscribe terraform-providers/terraform-provider-terraform
  2. /github unsubscribe terraform-providers/terraform-provider-terraform issues, deployments, statuses, public, commits,pulls

This should result in a final message saying:

This channel will get notifications from terraform-providers/terraform-provider-terraform for:

Why not use /github subscribe terraform-providers/terraform-provider-terraform releases? I've found that at least on mobile this fails to filter properly. For me, it ignored many filters. By doing it in two steps it worked every-time without issues.

This means you'll now get a nice formatted slack notification of just a release, without all the extra noise of pull requests, issues, commits and more. Easy way to keep up to date with projects if you want to keep it in slack. The other way is to use the github notifications section, but personally I like the slack approach.


Finally got atomic algolia to work in CICD for my hugo blog. I've tried tackling this in many ways over time, but this finally just worked with plug and play ease thanks to just adding a line to the hugo build netlify.toml file.

If you want to try this out, assuming you've already got an algolia index, json file generated and all... then just:

  1. Setup env variables in netlify build
  2. Add the following line to your netlify production build script
echo "Starting atomic-algolia update"
npm run algolia

You should get the following output from your netlify build if everything went right. No hits to algolia if you didn't change your indexes! ๐ŸŽ‰

3:13:47 PM: > sheldonhull.hugo@1.0.0 algolia /opt/build/repo
3:13:47 PM: > atomic-algolia
3:13:47 PM: [Algolia] Adding 0 hits to
3:13:47 PM: [Algolia] Updating 0 hits to
3:13:47 PM: [Algolia] Removing 0 hits from
3:13:47 PM: [Algolia] 156 hits unchanged in


Got some use out of the new beta plugins for netlify.

I added a link checker task and it ran over 2000 internal tests to ensure all internal resources such as images, internal links, and even css assests all correctly were found. ๐Ÿงน

That's a great idea for static sites. Incremental maintenance always is better than trying to catch up after years of no upkeep.

Using Pester to Improve Operational Tasks

Taking the Time To Test

Requirements in a constant state of change set you up for failure.

Failure to work through requirements before starting can also increase the risk of failure.

Planning with defined acceptance criteria of work you are committing to is not about filling in more documentation and work that no one cares about. Properly defining the acceptance criteria for yourself is about the exploratory process that defines and limits the scope of the work to deliver the minimum viable product. This allows continued momentum and delivery of value to the business.

Doesn't this create more busy work for me?

Without effective acceptance criteria, you are setting yourself up for more work, and thus likely to deliver value to the business as quickly. From my perspective, if you cannot truly define the acceptance criteria for work, then it's likely the effort will result in less value, be difficult to deliver, or end up with scope creep.

This is a critical thing to communicate to any teams struggling with reactive work. Without spending time ensuring proactive planning and defining the requirement and acceptance criteria, more time is often spent reworking and solving problems that might have been better handled with a little planning and forethought.

A classic example of unclear acceptance criteria and the wasteful impact of work is from a youtube clip here.

{{< youtube BTTdHW8Z668 >}}

How many times have you started work on a project and found yourself in a similar situation?

"There are many reasons why software projects go wrong. A very common reason is that different people in the organization or on the team have very different understandings of how the software should behave, and what problems itโ€™s trying to solve. Ignorance is the single greatest impediment to throughput" Teams that deliberately seek to discover what they are ignorant about before development starts are more productive because there is less rework. The most effective way to do this is through conversation and collaboration between key stakeholders... Dan North

Small Steps In Testing

Now that I did my high-level philosophizing ... let's do something more low level and fun.

An example of adding tests to DevOps oriented tasks

If we want to apply the "software engineer solving operational problems" approach to a very simple task, we could take the example of deploying a logging agent.

A software engineer expects a healthy culture to include tests for any code they write. This ensures better quality, more rapid velocity in changes, and other benefits that TDD advocates can tell you in other places.

For our example, let's say we are deploying a monitoring agent on some servers.


Gherkin is the syntax used to describe the tests.

{{< admonition type="info" title="Pester Syntax" >}} You can write Pester tests with PowerShell in a different format, but I'm going to use Gherkin here because of its sentences and far less of a confusing DSL for our example. {{< /admonition >}}

My understanding would be to compare Cucumber "spoken word", and Gherkin as the "language" chosen to communicate. When you write the feature files to work with Cucumber, you write them in Gherkin. Gherkin is supported for automated testing in many scripting languages. In a Windows environment, for example, you can use it to check many things like services running, connectivity with a server, installation of an app, and more using invocation via Pester.

Most examples I've found on Cucumber are focused very much on user testing, like website clicks, saving things, and other very narrow development focused actions.

What I've failed to see as much of is the discussion on the value of using this approach with teams implementing "Infrastructure as Code", operations teams, and other non-application specific roles.

Make Your Checklist Automated

In my example, let's start small and say you just have PowerShell, and some servers.

What I've discovered is that to actual validate DevOps oriented work is completed, you typically go through the equivalent of what a Cucumber test would have. This "checklist" of validations is often manually performed, lacking consistency and the ability to scale or repeat with minimal effort.

Consider an alternative approach to helping solve this issue, and expanding your ability to automate the tedious testing and validation of changes made.

Given (environment state)
When (something is done)
Then (expected result)

Using Pester To Write An Ops Test

Instead of using manual checks to validate a series of steps, let's see what a Cucumber oriented test might define as the acceptance criteria to confirm successfully finished, and how this might work with automating the tests.

This is a simple MSI with some command-line arguments that might be required. You want to validate the chocolatey package you deploy with correctly parsed the arguments passed in, and then successfully registered with the logging services.

An example feature file with Gherkin might look like this:

Feature: I can generate configuration files for FancyLoggingVendor collection dynamically

    Background: The test environment
        Given the test environment is setup

    Scenario: Source json is configured to collect windows event logs
        Given FancyLoggingVendor json object is initialized
        When the package parameters request a Windows Event
        Then the created json file contents match

Another one that shows Scenario Outlines. This adds more functionality to looping through various variations of test cases.

Feature: I can install FancyLoggingVendor on my machine

    Background: The local package is updated and the functions loaded
        Given the test environment is setup

    Scenario Outline: As a local user, I can install FancyLoggingVendor and configure capture my desired target
        Given this package is not installed
        When the package source is a local nupkg file
        And a <Description> is provided with the values: <Paths>
        Then I can install from this source
        And the sources json returns sources of: <ExpectedSourcesCount>
        And the application should show up in installed programs
        And the service should show up
        And the service should be running
        Examples: Source Variations
            | Description              | Paths                              | ExpectedSourcesCount |
            | Single_File              | C:\\temp\\taco.log                 | 1                    |
            | Single_File_with_Filters | C:\\temp\\_.log                    | 1                    |
            | SingleFolder             | C:\\temp\\foobar                   | 1                    |
            | MultipleFolders          | C:\\temp\\foobar,C:\\temp\\foobar2 | 2                    |
            | SingleFolder_with_filter | C:\\temp\\foobar\\_.log            | 1                    |

This provides us with a way to validate and test something like a chocolatey package installation or custom install script.

Leveraging Automation To Scale your Test

So all that writing...what benefit? Is it going to save time?

With PowerShell, I'd write something similar to match a called step to a code block here.

{{< admonition type="warning" title="Warning" >}}

Case sensitive name as of 2019-04-16 for keywords. Therefore to match steps, use Before not before or it won't match.

{{< /admonition >}}

# Uses PSFramework as greatly simplifies any variable scoping or general configuration work, making this type of test easily reusable
#this is incomplete, just to give some practical examples of how I might use to repeat a test like this.

Before Each Scenario {
# other prep work

Given 'local package is updated and the functions loaded' {
# package up the
Given 'the test environment is setup' {
# do setup stuff here or other stuff that might be needed
Given 'this package is not installed' {
{ choco uninstall (Get-PSFConfigValue 'gherkin.packagename') } | Should -Not -Throw
When 'the package source is a local nupkg file' {
Set-PSFConfig 'gherkin.packagesource' -Value (Get-ChildItem -Path (Get-PSFConfigValue 'gherkin.packagefolder') -Filter *.nupkg).FullName

Then 'I can install from this source' {

$ChocoArguments = @()
$ChocoArguments += 'upgrade'
$ChocoArguments += Get-PSFConfigValue 'gherkin.packagename'
$ChocoArguments += '--source "{0}"' -f (Get-PSFConfigValue 'gherkin.packagefolder')
$ChocoArguments += (Get-PSFConfigValue 'gherkin.params')
$ChocoArguments += '--verbose'
{ Start-Process choco.exe -ArgumentList $ChocoArguments -NoNewWindow -PassThru | Wait-Process } | Should -Not -Throw

And 'the sources json returns sources of: <ExpectedSourcesCount>' {
Write-PSFmessage -Level Important -FunctionName 'FancyLoggingVendor.Install.steps.ps1' -Message "ExpectedSourcesCount -- $ExpectedSourcesCount"
$DefaultJsonFile = 'PathToJsonHere'
Test-Path $DefaultJsonFile -PathType Leaf | Should -Be $True
@(Get-Content $DefaultJsonFile -Raw | ConvertFrom-Json | Select-Object Sources).Count | Should -Be $ExpectedSourcesCount
And 'the application should show up in installed programs' {

# helper function could be loaded and used to parse registry info for installed app showing up, or you could code it directly

Get-InstalledApp 'MyAgent' | Should -Be $true
And 'the service should show up' {
@(Get-Service 'MyAgent*').Count | Should -Be 0
And 'the service should be running' {
@(Get-Service 'MyAgent*' | Where-Object Status -eq 'Running').Count | Should -Be 0

Now that I wrote this, I could simply run:

Invoke-Gherkin -Path InstallFancyLoggingVendor.steps.ps1

This would run all the steps from a feature file and ensure I'm able to repeat these tests after any change to confirm they work.

Other Great Use Cases

I've leveraged this to validate SQL Server configuration changes on a new AWS RDS Deployment, validate build steps completed successfully, tested file paths existing, and more. I really like how you can have this all integrated in a nice UI by uploading the nunit tests in Azure DevOps pipelines too.

Start Small

Take a look at the simple Pester syntax examples or the gherkin examples I gave and use that to do anything you keep having to check more than a few times. You'll find your efforts rewarded by having more consistent testing and probably save quite a bit of effort as well.


If you found these concepts helpful and would like an example of using Pester to test SQL Server login authentication, user group assignment, and more, let me know. I've done SQL Server pester tests using traditional Pester syntax that would validate maintenance solution deployment, login mapping, and more. If this was valuable, I could do a write-up of this.

If you also would like more fundamentals, I'd be willing to do a basic Pester write-up for an operational focused task that is ground zero too, just depends if you the reader find this helpful.

Comments always appreciated if this helped you! Let's me know it's actually helping someone out and always great to connect with others. ๐Ÿป

Improved Windows Terminal Setup

I've long been a Cmder/ConEmu user for Windows as it's provided a much-needed improvement of the standard Windows terminal.

I've started trying to use the newer Windows Terminal project to benefit from the improved performance and support, and found getting it up and running with any customizations was a little time consuming and confusing. This wasn't something I'd hand off to someone who wasn't looking for experimentation.

So here it goes! Rather than hunting all around to get a nice start on some default prompts, I've linked to some gists that can help you get up and running quickly with a few extra perks.

Getting Started

This will help you get the terminal installed, along with downloading some settings I've already pre-setup with keybindings and more.

To customize your own keybindings, you can go to the profiles.json documentation.

{{< gist sheldonhull "93d8060e6f86e0c46535ef6699d6e0c8" "install-settings.ps1" >}}

Improve Your Experience

After install, you can run the next command to help you get a better font setup with full support for ligatures and more.

{{< gist sheldonhull "93d8060e6f86e0c46535ef6699d6e0c8" "install-font.ps1" >}}

And after this, if you don't have a nice PowerShell prompt experience, this will help give you a great start. This contains a few things, including starship. This is really useful as it has a library of prompt enhancements baked in. For example, if you are have an AWS profile active, it will display that for reference. It can display an active terraform workspace, git branch info, python virtual environment and more. Definitely a nice quick productivity booster with no real configuration needed to get going.

{{< gist sheldonhull "93d8060e6f86e0c46535ef6699d6e0c8" "configure-profile-prompt.ps1" >}}

If you are wondering why I didn't leave the pretty awesome "useAcrylic": true on for my main pwsh session, it's because I found the background contrast reduction made it hard to read some darker colors on the prompt.

Be sure to try out the retro pwsh theme for some nice eye candy.

The result

PowerShell Protip: Note the suggested completion based on prior commands in the pwsh prompt. This is some great prerelease work on a better PSReadline experience with Powershell.

image of windows terminal

images of windows terminal

A Smattering of Thoughts About Applying Site Reliability Engineering principles

What's This about

I figured I'd go ahead and take this article which I've gutted several times and share some thoughts, even if it's not an authority on the topic. ๐Ÿ˜€

In the last year, I've been interested in exploring the DevOps philosophy as it applies to operations as well as software development. I originally started various drafts on these concepts back before April 2019, but never got around to finishing it until now.

This isn't a very focused post, more a smattering of thoughts, so let's get to it! ๐Ÿ’ช

DevOps Is A Challenge

Having come from a development background, applying DevOps to the operational side of the fence is an interesting challenge.

There are so many things that can be unique to the evolving way a team you are part of is growing and learning, that it can be difficult sometimes to know what is "best practice" and was is "essential practice" in a DevOps culture.

  • What does it mean to plan and scale your workflow in a DevOps culture?
  • How do operational stability and the pressure for new feature delivery in a competitive market meet in a healthy tension?
  • How do you do "everything-as-code" in practice?
  • Lots more!

There are various ways to implement DevOps, and core principles to DevOps. I've found that one that really resonated with me as I've looked for how others have done this.

Site Reliability Engineering

Google's Site Reliability Engineering (SRE) material provides a solid guide on the implementation of SRE teams. They consider SRE and DevOps in a similar relationship to how Agile views Scrum. Scrum is an implementation of Agile tenants. Site Reliability Engineering is an implementation of a DevOps culture, with guidelines on how to approach this in a systematic way.

If you think of DevOps like an interface in a programming language, class SRE implements DevOps. Google SRE

What I like about the material, is that a lot of the fixes I've considered to improvements in workflow and planning have been thoughtfully answered in their guide, since it's a specific implementation rather than a broader philosophy with less specific steps.

Regardless of where you are in your journey, a lot of the principles have much to offer. Even smaller organizations can benefit from many of these concepts, with various adjustments being made to account for the capabilities of the business.

Recommended reading if this interests you:

Deploys: Itโ€™s Not Actually About Fridays โ€“
DevOps Topologies
Do you have an SRE team yet? How to start and assess your journey
Google - Site Reliability Engineering
Love (and Alerting) in the Time of Cholera (and Observability) โ€“

Where I Started

At the beginning of my tech career, I worked at a place that the word "spec" or "requirement" was considered unreasonable. Acceptance criteria would have been looked down upon, as something too formal and wasteful.

While moving towards to implementation of any new project, I was expected to gather the requirements, build the solution, ensure quality and testing, and deploy to production. That "cradle to grave" approach done correctly can promote the DevOps principles, such as ensuring rapid feedback from the end-users and ownership from creation through to production.

A Different Take

I've been in a somewhat interesting blend of roles that gives me some insight into this. As a developer, I always look at things from an ๐Ÿค– automation & coding perspective. It's the exception, rather than the norm for me to do it manually without any type of way to reproduce via code.

I've also been part of a team that did some automation for various tasks in a variety of ways, yet often resolved issues via manual interactions due to the time constraints and pressures of inflowing work. Building out integration tests, code unit tests, or any other automated testing was a nice idea in theory, but allocating time to slow down and refactor for automated testing on code, deployments, and other tasks were often deemed too costly or time prohibitive to pursue, in addition to requiring a totally different skillset to focus on.

For example, you can't write code based tests against a deployment, unless you have made the time and effort to learn to code in that language and work through the pain of discovery in writing integration oriented tests. It's not a simple feat ๐Ÿฆถ to just pick up a language and start writing tests in it if you've never done this before.


Reactive work also challenges DevOps focused teams that do operational work.

  • Difficult to categorize emergency from something that could be done in a few weeks
  • Difficult to deliver on a set of fully completed tasks in a sprint (if you even do a sprint)
  • High interrupt ratio for request-driven work instead of able to put into a sprint with planning. (This is common in a DevOps dedicated team topology that is in some of the Anti-Types mentioned in DevOps Anti-Type topologies)
  • Sprint items in progress tend to stay there for more than a few days due to the constant interruptions or unplanned work items that get put on their plate.
  • Unplanned work items are constant, going above the 25% buffer normally put into a sprint team.
  • Continued feature delivery pressure without the ability to invest in resiliency of the system.

Google has a lot more detail on the principles of "on-call" rotation work compared to project-oriented work. Life of An On-Call Engineer. Of particular relevance is mention of capping the time that Site Reliability Engineers spend on purely operational work to 50% to ensure the other time is spent building solutions to impact the automation and service reliability in a proactive, rather than reactive manner.

In addition, the challenges of operational reactive work and getting in the zone on solving project work with code can limit the ability to address the toil of continual fixes. Google's SRE Handbook also addresses this in mentioning that you should definitely not mix operational work and project work on the same person at the same time. Instead whoever is on call for that reactive work should focus fully on that, and not try to do project work at the same time. Trying to do both results in frustration and fragmentation in their effort.

This is refreshing, as I known I've felt the pressure of needing to deliver a project, yet feeling that pressure of reactive work with operational issues taking precedence.

It's important to recognize the importance of that operational work. It helps ensure a team is delivering what is most important to a business, a happy customer base! At the same time, always living in the reactive world can easily cause the project related work and reliability improvements that will support those efforts to suffer if a intentional plan to handle this is not made.

Reducing Toil

I've always been focused on building resilient systems, sometimes to my own detriment velocity wise. Balancing the momentum of delivery features and improving reliability is always a tough issue to tackle. Automation isn't free. It requires effort and time to do correctly. This investment can help scaling up what a team can handle, but requires slower velocity initially to do it right.

How do you balance automating and coding solutions to manual fixes, when you often can't know the future changes in priority?

It can be applied to immediate needs

A leader I've worked with mentioned one great way is to ensure whatever solution you build is immediately applicable to solving work in progress. By this, he meant work that could immediately benefit a team, rather than building things for future possibilities that haven't yet come into play. This aligns with a LEAN mentality and there is a lot to like about it.

It solves an immediate pain

Another way I've begun looking at this is solving the key pain-points that are mentioned by others. I tend to look at many improvements, and it can be difficult to choose what area to tackle at times, as I can see value in many of them.

However, solving paint points that are directly impacting someone enough that they are asking for help should take precedence in many cases (not all).

Why? This pain-point solved gains you an immediate win.

If you are solving an issue that others are ok with, or don't recognize as an issue, then you have an uphill battle for adoption compared to solving something immediately impacting their work. This doesn't negate addressing problems that others have not yet recognized, it is just something that's become clear can help in moving forward with improvements with less resistance.

Last Priority - Solving Problems No One Yet Knows They Need Solved

There is an important caveat to all of this. There is a realm of problems when you have expertise in an area that you will identify that others don't see. When assessed against goals of a business, these can be critical.

In my experience, the only way for these identified issues to be clearly prioritized is having a strong ability to communicate the problem and design some clear objectives to place this into the team backlog to address.

Verbally communicating, while important, won't have any staying power compared to a clearly defined objective in a backlog and advocated for in a presentation that helps raise concern for the same issue you care about.

They key is that your signal-to-noise-ratio is good, and when you raise the concern about the issue others don't have to work hard to understand why it's a problem.

How to balance product vs reliability improvements

SREs balance the juggling of product velocity and reliability improvements in a unique way... the usage of SLO and the error budget concept.

SLO & Error Budgets

The concepts of having an error budget really struct home for me. Clear Service Level Objectives (SLO) and error budgets to work with helps ensure both the development velocity and stability desired by operations is handled correctly.

Error budgets provide the key ingredient to balancing new feature delivery, while still ensuring happy customers with a stable system.

One of the best sections I've read on this was: Embracing Risk.

Error budgets are discussed, and internal SLI (Server Level Indicators). These are integral to ensuring a balance of engineering effort in balance with new feature delivery. The goal of 100% reliability, while sounding great, is inherently unrealistic.

Product development performance is largely evaluated on product velocity, which creates an incentive to push new code as quickly as possible. Meanwhile, SRE performance is (unsurprisingly) evaluated based upon the reliability of a service, which implies an incentive to push back against a high rate of change. Information asymmetry between the two teams further amplifies this inherent tension.

Wrap Up

I've been mulling over this for a while, and find a lot of the concepts very helpful as I've been challenged with both operational and development focus. As always, these are personal thoughts and don't reflect the opinion of my employer. I hope it gives you some food for thought. If you have some feedback, just post up a comment and let's dialogue!

Quickly Create Github Release via Cli


I've been trying to improve modularization with Terraform. Using Terraform Cloud, you get a private module repository. The modules are linked to tags in git.

I've wanted additionally to specifically create releases for the versions in addition to the tags, to make tracking of usable versions a little cleaner.

There are several ways to do this, including using the GitHub API, npm modules, and more. I wanted a simple CLI tool to do this and ran across this great Go utility that worked seamlessly for me.

I've used the Hub cli but the create release syntax never really worked for me.


Github-release is a simple golang cli that worked great for me. Note that this is a fork. This fork is more up to date than the original.

With go installed just run this to get it installed and available in PATH.

go get

To simplify GitHub access, ensure you set an environment variable for your user called GITHUB_TOKEN.

With PowerShell you can do it quickly like this (you might need to close and reopen vscode/terminal for this to be recognized)



To use this, you can chain together some steps and see how it can save you time on creating a GitHub release.

{{< gist sheldonhull "53055bbff368a4ebe4e0794076a56c37" >}}

This helped me get moving faster โšก on using Github releases without the tedious work to create. If this helped you or have any feedback, drop a comment below and let me know! The comments are powered by Utterances which will open a Github issue to discuss further. ๐Ÿ‘

Asciidoc Distributed Docs as Code

The Problem

  • I want to keep my code and my documentation in the same place.
  • I want to separate the presentation layer from my documentation content.
  • I want to be flexible to publish documentation to a variety of endpoints and formats as processes evolve, without my content being impacted.
  • I want to write a solid runbook for things that can't be fully automated, but still include scripts and other code in their native format.

Documentation is such an important part of a developer's life. I think we often take it for granted, and it's an afterthought in many projects. However, as I consider my work, I know that I'm not reinventing the wheel very often ๐Ÿ˜€. Most of what I do is built on the back of others' work. When I use tooling, I'm reading the documentation and using it as my basis to get work done. When I use my notes and blog posts as a reference, I'm using my informal version of knowledge gathering.

INVEST in documenting your work as you go, for the person behind you. You don't find time to do it, you make time to do it while you work, as a first class citizen of your work, not an after-thought. Think of all the times you've had to dig for answers and save someone else that experience.

You code and document not as much for yourself, but for the person that comes behind you.


I've found a happy solution in the Asciidoctor documentation format over markdown. You can go google this for more expanded understanding, but I've decided that other than for basic notes and blog posts which are very simplistic, I now choose Asciidoctor.

Why use Asciidoc format over markdown comes down to the needs of technical documentation.

Here are some key reasons why I've found Asciidoc format to be worth learning:

  • I can reference code files with a simple include::file[] statement, while markdown would require me to embed my code directly as a code block.
  • I can generate a table from a csv file, further helping me automate a refresh of the underlying data that is rendered to a table display
  • I can create tables much more cleanly and with control than in markdown, even allowing nested tables for complicated process documentation.
  • Automatic admonition callouts without extensions using simple statements like IMPORTANT: foo


Since the common documentation system used where I am at is Confluence, I decided to leverage the incredible confluence-publisher project that made this entire process a breeze. Check the repo and the linked documentation out here: Confluence Publisher

In the future, if I didn't use confluence, I'd explore rendering as a static website through Hugo (that's what this site is generated from) or revisit Antora and maybe merge my content into the format required by Atora programmatically.

Use Docker

Since Asciidoc is written in Ruby, use docker and you won't have to deal with dependency nightmares, especially on Windows.

$RepoDirectoryName = 'taco-ops-docs'
echo "๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ๐ŸŒฎ"
echo "Running confluence publisher ๐ŸŒฎ"
echo "๐Ÿ“ƒ Publishing $RepoDirectoryName repo contents"

docker run --rm -v $BUILD_SOURCESDIRECTORY/$RepoDirectoryName/docs:/var/asciidoc-root-folder -e ROOT_CONFLUENCE_URL=$ROOT_CONFLUENCE_URL \
echo "๐Ÿ“ƒ Publishing $RepoDirectoryName repo contents finished"

Yes... I know. I get bored reading log messages when debugging so my new year premise was to add some emoji for variety. Don't judge. ๐Ÿ˜

Distributed Docs Structure

So the above approach is fantastic for a single repo. I wanted to take it to a different level by solving this problem for distributed documentation. By distributed I meant that instead of containing all the documentation in a single "wiki" style repo, I wanted to grab documentation from the repositories I choose and render it. This would allow the documentation related to being contained in the repository it is related to.

For instance, what if I wanted to render the documentation in the following structure:

** General Documentation**
---> building-tacos
--------> topic.adoc
---> eating-tacos
--------> topic.adoc
---> taco-policies
--------> topic.adoc
---> taco-as-code
--------> topic.adoc

** Repo Oriented Documentation**
---> taco-migration
--------> category-1
------------> topic.adoc
------------> topic.adoc
--------> category-2
------------> topic.adoc
------------> topic.adoc
---> taco-monitoring
--------> category-1
------------> topic.adoc
------------> topic.adoc
--------> category-2
------------> topic.adoc
------------> topic.adoc

The only current solution found was Antora. Antora is very promising and great for more disciplined documentation approaches by software development teams. The limitation I faced was complexity and rigidity in structure. For Antora to generate a beautiful documentation site, you have to ensure the documentation is structured in a much more complex format. For example, the docs might be under docs/modules/ROOT/pages/doc.adoc and have a nav.adoc file as well. While this promises a solid solution, retrofitting or expecting adoption might be tricky if your team has never even done markdown.

Azure DevOps Pipeline

I ended using an Azure DevOps pipeline (YAML of course ๐Ÿค˜) that provides a nice easy way to get this done.

First, for proper linking, you should follow the directions Azure DevOps gives on the creation of a Github Service Connection which uses OAUTH. This will ensure your setup isn't brittle and using your access token.

{{< gist sheldonhull "053cb176d5c2847a4e323f01207acb82" >}}

Things to Know

  • Ensure you use the format shown here for documentation to render in confluence correctly. You need to have the names match in the doc/folder for it to know to render the child pages.
** Repo Oriented Documentation**
README.adoc  -- optional, but I always include for landing page, and point to the docs folder using link:./docs/myrepo.adoc
---> [docs]
------> [resources]  -- optional, but keeps the scripts organized and consistent, or any images
------> process.adoc
------> another-process.adoc
---> taco-ops-repo.adoc
  • Include your scripts by using include::./resources/_myscript.ps1[]. You may have to test that relative path issue if doing multiple repos.
  • Ensure your non-asciidoc contents are prefaced with an underscore in the title name. I don't like this, but it's a requirement from confluence-publisher. This ensures it won't try to render as a page.
  • Anything in the target directory (ancestor) gets purged in the process. I recommend a dedicated confluence space you create just for this to minimize risk and disable manual edits.

{{< admonition type="info" title="Docker Commands in Microsoft-Hosted Agent" >}} I didn't expect docker commands to work in Azure DevOps agent, thinking nested virtualization would not work and all. However, it works beautifully. Consider using Azure DevOps yaml pipelines for running your docker commands and you take one step towards better build processes. {{< /admonition >}}

Running SQL Server in Docker

{{< admonition type="info" title="Updated 2020-05-05" >}}

I've had lots of challenges in getting docker for sql-server working because I've wanted to ensure for my dev use case that there was no need for virtual volume management and copying files into and out of this. Instead, I've wanted to bind to local windows paths and have it drop all the mdf/ldf right there, so even on container destruction everything is good to go.

After working through the changes in SQL 2019 that require running as non-root, I've gotten it work again. No install of sql-server needed. Easy disposable development instance through docker! I'll update my docker compose content when I can, but in the meantime, this should get you running even more quickly with SQL Server 2019.

docker run `
    --name SQL19 `
    -p 1433:1433 `
    -e "ACCEPT_EULA=Y" `
    -e "MSSQL_SA_PASSWORD=ThisIsNotARealPassword@!1}" `
    -v C:\mssql\SQL19:/sql `

docker run `
    --name SQL19WithSpaces `
    -p 1434:1433 `
    -e "ACCEPT_EULA=Y" `
    -e "MSSQL_SA_PASSWORD=ThisIsNotARealPassword@!1}" `
    -v C:\mssql\SQL19WithSpaces:/sql `

{{< /admonition >}}

Why Use Docker for MSSQL

Say you have an onboarding and need to get a new developer up to speed. Have you gone through the setup process for SQL Server on Windows? It's a bit tedious and even with running scripts to install (or in my case I wrapped up with Chocolatey), it's still a lot of things you need, including possibly a reboot if missing some .net dependencies.

The normal solution is to leverage more localdb for quickly deploying and isolating the database.

This has its own set of challenges that personally I like to avoid. Localdb is more isolated, but it's not truly like running a local SQL Server standard edition, and debugging permissions, sharing, and other things can with its own set of steps to work through. I've seen it common for many devs to just avoid these issues and run a local SQL Server installation to just simplify and work with SQL Server as it's expected to be.

I'd explored Docker SQL Server containers in the past, but one big issue for adoption to me was the issues I had mounting the local Windows folders to the Linux based container. โ“ Why is this important? Ease of usage for a developer. If I proposed this would save effort to many developers working with SQL Server, I'm pretty sure telling them that they'd have to copy a backup or database file in via CLI to a virtual mounted drive that they can't easily manage would be a deal-breaker. I basically wanted to ensure if they spun up SQL Server in a container, the database backups, mdf and ldf files, and any other created files would be able to persist outside of that instance, ensuring easy development workflow.

I'm happy to say I finally have it working to my satisfaction, resolving most of those concerns.

Scenarios This is Good For

If you've desired to do the following, then using Docker might end up saving you some effort.

  • simplify the setup of a new SQL Server instance
  • be able to reset your SQL Server instance to allow testing some setup in isolation
  • be able to switch SQL Server editions to match a new requirement
  • be able to upgrade or patch to a later version of SQL Server with minimal effort

This is not for production. There's more effort and debate that goes on to using SQL Server in containers, Kubernetes (even more complex!), and other containerization approaches that are far outside the scope of what I'm looking at. This is first and foremost focused on ensuring a development workflow that reduces complexity for a developer and increases the ease at which SQL Server testing can be implemented by making the SQL Server instance easily disposed of and recreated.

If this also means I spread some ๐Ÿง Linux magic to some Windows stack developers... then count it as a sneaky win. ๐Ÿ˜

SQL Server on Linux or Windows containers

The following example is done with SQL Server on Linux. As I'm already comfortable with SQL Server on Windows, I wanted to be try this on Linux based container. I also wanted to continue using Linux based containers for tooling, and not have to switch back to Windows containers for the sole purpose of running SQL Server. At the time I began this testing, I found it was exclusive. You either ran Linux or Windows-based containers. This is changing with the advent of new features in Docker that are there to allow side by side Windows + Linux based containers.

Release notes indicate:

Experimental feature: LCOW containers can now be run next to Windows containers (on Windows RS3 build 16299 and later). Use --platform=linux in Windows container mode to run Linux Containers On Windows. Note that LCOW is experimental; it requires the daemon experimental option. Docker Community Edition 18.03.0-ce-win59 2018-03-26

The main difference in your local development process will be Windows Authentication vs SQL Authentication. Use SQL Authentication with Linux based SQL Server

Docker Compose Example

The following is the result of a lot of trial and error over a year. Thanks to Shawn Melton ๐Ÿ‘ also for providing me with support doing my troubleshooting as part of the SQL Server community in Slack. You can find Shawn's example I used as a starting point for my ๐Ÿงชtesting in this gist.

A few helpful tips:

  1. Remove -d for detached and you can see the SQL Server console output in the console.
  2. See the persisted databases (system and user!) in the artifacts directory after docker-compose begins running.

{{< gist sheldonhull a70a3a731b329b67f47a331c64c72ab5 >}}

Improving Code Tests

Another reason I'm really excited about is the ability to better support testing through tools like Pester. Yes, it's a geek thing, but I love a solid written test ๐Ÿงชthat maintains my code and helps me ensure quality with changes. Better TDD is a goal of mine for sure.

๐Ÿ”จ This supports implementation of better tests by providing the potential for spinning up a local SQL Instance, restoring a test database, and running a sequence of actions against it with pass or fail without the additional infrastructure requirements to have this done on another server. Making your tests that are not full integration testing as minimally dependent on external factors is a fantastic step to saving you a lot of work.

A simple pester (PowerShell) might frame the start of a test like this:

Before All {
    docker-compose up -d
    Import-Module Dbatools
    # Wait Until dbatools confirms connectivity through something like test-dbaconnection, then proceed with tests
    # Test-DbaConnection ....
    # Restore-DbaDatabase ...

After All {
    docker-compose down --volume

Describe "DescribeName" {
    Context "ContextName" {
        It "ItName" {

Wrap Up

Hope this helps someone. I spent at least a year coming back over time to this hoping to actually get it working in a way that felt like a first-class citizen and reduced complexity for development work.

I'm pretty happy with the results. ๐Ÿ˜