Kept getting an error Incorrect syntax near the keyword 'read' when running the some updated PowerShell 7 in lambda with the dotnet3.1 sdk. Was troubleshooting loading types thinking I was having path issues.
Turns out one of these things is not like the other. 🤣
I think this explains why "error during "read". Maybe I should have run in my Azure Data Studio session before trying serverless 😀
If you want to simplify keeping up with github releases instead of using email, try the GitHub slack app. Simple create a releases channel and then run these two steps:
/github subscribe terraform-providers/terraform-provider-terraform/github unsubscribe terraform-providers/terraform-provider-terraform issues, deployments, statuses, public, commits,pullsThis should result in a final message saying:
This channel will get notifications from terraform-providers/terraform-provider-terraform for:
releases
Why not use /github subscribe terraform-providers/terraform-provider-terraform releases? I've found that at least on mobile this fails to filter properly. For me, it ignored many filters. By doing it in two steps it worked every-time without issues.
This means you'll now get a nice formatted slack notification of just a release, without all the extra noise of pull requests, issues, commits and more. Easy way to keep up to date with projects if you want to keep it in slack. The other way is to use the github notifications section, but personally I like the slack approach.
Finally got atomic algolia to work in CICD for my hugo blog. I've tried tackling this in many ways over time, but this finally just worked with plug and play ease thanks to just adding a line to the hugo build netlify.toml file.
If you want to try this out, assuming you've already got an algolia index, json file generated and all... then just:
You should get the following output from your netlify build if everything went right. No hits to algolia if you didn't change your indexes! 🎉
3:13:47 PM: > sheldonhull.hugo@1.0.0 algolia /opt/build/repo
3:13:47 PM: > atomic-algolia
3:13:47 PM: [Algolia] Adding 0 hits to sheldonhull.com
3:13:47 PM: [Algolia] Updating 0 hits to sheldonhull.com
3:13:47 PM: [Algolia] Removing 0 hits from sheldonhull.com
3:13:47 PM: [Algolia] 156 hits unchanged in sheldonhull.com
Got some use out of the new beta plugins for netlify.
I added a link checker task and it ran over 2000 internal tests to ensure all internal resources such as images, internal links, and even css assests all correctly were found. 🧹
That's a great idea for static sites. Incremental maintenance always is better than trying to catch up after years of no upkeep.
Requirements in a constant state of change set you up for failure.
Failure to work through requirements before starting can also increase the risk of failure.
Planning with defined acceptance criteria of work you are committing to is not about filling in more documentation and work that no one cares about. Properly defining the acceptance criteria for yourself is about the exploratory process that defines and limits the scope of the work to deliver the minimum viable product. This allows continued momentum and delivery of value to the business.
Without effective acceptance criteria, you are setting yourself up for more work, and thus likely to deliver value to the business as quickly. From my perspective, if you cannot truly define the acceptance criteria for work, then it's likely the effort will result in less value, be difficult to deliver, or end up with scope creep.
This is a critical thing to communicate to any teams struggling with reactive work. Without spending time ensuring proactive planning and defining the requirement and acceptance criteria, more time is often spent reworking and solving problems that might have been better handled with a little planning and forethought.
A classic example of unclear acceptance criteria and the wasteful impact of work is from a youtube clip here.
{{< youtube BTTdHW8Z668 >}}
How many times have you started work on a project and found yourself in a similar situation?
"There are many reasons why software projects go wrong. A very common reason is that different people in the organization or on the team have very different understandings of how the software should behave, and what problems it’s trying to solve. Ignorance is the single greatest impediment to throughput" Teams that deliberately seek to discover what they are ignorant about before development starts are more productive because there is less rework. The most effective way to do this is through conversation and collaboration between key stakeholders... Dan North
Now that I did my high-level philosophizing ... let's do something more low level and fun.
If we want to apply the "software engineer solving operational problems" approach to a very simple task, we could take the example of deploying a logging agent.
A software engineer expects a healthy culture to include tests for any code they write. This ensures better quality, more rapid velocity in changes, and other benefits that TDD advocates can tell you in other places.
For our example, let's say we are deploying a monitoring agent on some servers.
Gherkin is the syntax used to describe the tests.
{{< admonition type="info" title="Pester Syntax" >}} You can write Pester tests with PowerShell in a different format, but I'm going to use Gherkin here because of its sentences and far less of a confusing DSL for our example. {{< /admonition >}}
My understanding would be to compare Cucumber "spoken word", and Gherkin as the "language" chosen to communicate. When you write the feature files to work with Cucumber, you write them in Gherkin. Gherkin is supported for automated testing in many scripting languages. In a Windows environment, for example, you can use it to check many things like services running, connectivity with a server, installation of an app, and more using invocation via Pester.
Most examples I've found on Cucumber are focused very much on user testing, like website clicks, saving things, and other very narrow development focused actions.
What I've failed to see as much of is the discussion on the value of using this approach with teams implementing "Infrastructure as Code", operations teams, and other non-application specific roles.
In my example, let's start small and say you just have PowerShell, and some servers.
What I've discovered is that to actual validate DevOps oriented work is completed, you typically go through the equivalent of what a Cucumber test would have. This "checklist" of validations is often manually performed, lacking consistency and the ability to scale or repeat with minimal effort.
Consider an alternative approach to helping solve this issue, and expanding your ability to automate the tedious testing and validation of changes made.
Instead of using manual checks to validate a series of steps, let's see what a Cucumber oriented test might define as the acceptance criteria to confirm successfully finished, and how this might work with automating the tests.
This is a simple MSI with some command-line arguments that might be required. You want to validate the chocolatey package you deploy with correctly parsed the arguments passed in, and then successfully registered with the logging services.
An example feature file with Gherkin might look like this:
@class
Feature: I can generate configuration files for FancyLoggingVendor collection dynamically
Background: The test environment
Given the test environment is setup
Scenario: Source json is configured to collect windows event logs
Given FancyLoggingVendor json object is initialized
When the package parameters request a Windows Event
Then the created json file contents match
"""
{
"api.version":"v1",
"sources":\[
{
"sourceType":"LocalWindowsEventLog",
"name":"GherkinTest",
"renderMessages":true,
"cutoffRelativeTime":"-1h",
"hostname":"gherkintest",
"logNames":\[
"Security",
"Application"
\]
}
\]
}
"""
Another one that shows Scenario Outlines. This adds more functionality to looping through various variations of test cases.
Feature: I can install FancyLoggingVendor on my machine
Background: The local package is updated and the functions loaded
Given the test environment is setup
Scenario Outline: As a local user, I can install FancyLoggingVendor and configure capture my desired target
Given this package is not installed
When the package source is a local nupkg file
And a <Description> is provided with the values: <Paths>
Then I can install from this source
And the sources json returns sources of: <ExpectedSourcesCount>
And the application should show up in installed programs
And the service should show up
And the service should be running
Examples: Source Variations
| Description | Paths | ExpectedSourcesCount |
| Single_File | C:\\temp\\taco.log | 1 |
| Single_File_with_Filters | C:\\temp\\_.log | 1 |
| SingleFolder | C:\\temp\\foobar | 1 |
| MultipleFolders | C:\\temp\\foobar,C:\\temp\\foobar2 | 2 |
| SingleFolder_with_filter | C:\\temp\\foobar\\_.log | 1 |
This provides us with a way to validate and test something like a chocolatey package installation or custom install script.
So all that writing...what benefit? Is it going to save time?
With PowerShell, I'd write something similar to match a called step to a code block here.
{{< admonition type="warning" title="Warning" >}}
Case sensitive name as of 2019-04-16 for keywords. Therefore to match steps, use Before not before or it won't match.
{{< /admonition >}}
# Uses PSFramework as greatly simplifies any variable scoping or general configuration work, making this type of test easily reusable
#this is incomplete, just to give some practical examples of how I might use to repeat a test like this.
Before Each Scenario {
# other prep work
}
Given 'local package is updated and the functions loaded' {
# package up the
}
Given 'the test environment is setup' {
# do setup stuff here or other stuff that might be needed
}
Given 'this package is not installed' {
{ choco uninstall (Get-PSFConfigValue 'gherkin.packagename') } | Should -Not -Throw
}
When 'the package source is a local nupkg file' {
Set-PSFConfig 'gherkin.packagesource' -Value (Get-ChildItem -Path (Get-PSFConfigValue 'gherkin.packagefolder') -Filter *.nupkg).FullName
}
Then 'I can install from this source' {
$ChocoArguments = @()
$ChocoArguments += 'upgrade'
$ChocoArguments += Get-PSFConfigValue 'gherkin.packagename'
$ChocoArguments += '--source "{0}"' -f (Get-PSFConfigValue 'gherkin.packagefolder')
$ChocoArguments += (Get-PSFConfigValue 'gherkin.params')
$ChocoArguments += '--verbose'
{ Start-Process choco.exe -ArgumentList $ChocoArguments -NoNewWindow -PassThru | Wait-Process } | Should -Not -Throw
}
And 'the sources json returns sources of: <ExpectedSourcesCount>' {
param($ExpectedSourcesCount)
Write-PSFmessage -Level Important -FunctionName 'FancyLoggingVendor.Install.steps.ps1' -Message "ExpectedSourcesCount -- $ExpectedSourcesCount"
$DefaultJsonFile = 'PathToJsonHere'
Test-Path $DefaultJsonFile -PathType Leaf | Should -Be $True
@(Get-Content $DefaultJsonFile -Raw | ConvertFrom-Json | Select-Object Sources).Count | Should -Be $ExpectedSourcesCount
}
And 'the application should show up in installed programs' {
# helper function could be loaded and used to parse registry info for installed app showing up, or you could code it directly
Get-InstalledApp 'MyAgent' | Should -Be $true
}
And 'the service should show up' {
@(Get-Service 'MyAgent*').Count | Should -Be 0
}
And 'the service should be running' {
@(Get-Service 'MyAgent*' | Where-Object Status -eq 'Running').Count | Should -Be 0
}
Now that I wrote this, I could simply run:
This would run all the steps from a feature file and ensure I'm able to repeat these tests after any change to confirm they work.
I've leveraged this to validate SQL Server configuration changes on a new AWS RDS Deployment, validate build steps completed successfully, tested file paths existing, and more. I really like how you can have this all integrated in a nice UI by uploading the nunit tests in Azure DevOps pipelines too.
Take a look at the simple Pester syntax examples or the gherkin examples I gave and use that to do anything you keep having to check more than a few times. You'll find your efforts rewarded by having more consistent testing and probably save quite a bit of effort as well.
If you found these concepts helpful and would like an example of using Pester to test SQL Server login authentication, user group assignment, and more, let me know. I've done SQL Server pester tests using traditional Pester syntax that would validate maintenance solution deployment, login mapping, and more. If this was valuable, I could do a write-up of this.
If you also would like more fundamentals, I'd be willing to do a basic Pester write-up for an operational focused task that is ground zero too, just depends if you the reader find this helpful.
Comments always appreciated if this helped you! Let's me know it's actually helping someone out and always great to connect with others. 🍻
I've long been a Cmder/ConEmu user for Windows as it's provided a much-needed improvement of the standard Windows terminal.
I've started trying to use the newer Windows Terminal project to benefit from the improved performance and support, and found getting it up and running with any customizations was a little time consuming and confusing. This wasn't something I'd hand off to someone who wasn't looking for experimentation.
So here it goes! Rather than hunting all around to get a nice start on some default prompts, I've linked to some gists that can help you get up and running quickly with a few extra perks.
This will help you get the terminal installed, along with downloading some settings I've already pre-setup with keybindings and more.
To customize your own keybindings, you can go to the profiles.json documentation.
{{< gist sheldonhull "93d8060e6f86e0c46535ef6699d6e0c8" "install-settings.ps1" >}}
After install, you can run the next command to help you get a better font setup with full support for ligatures and more.
{{< gist sheldonhull "93d8060e6f86e0c46535ef6699d6e0c8" "install-font.ps1" >}}
And after this, if you don't have a nice PowerShell prompt experience, this will help give you a great start. This contains a few things, including starship. This is really useful as it has a library of prompt enhancements baked in. For example, if you are have an AWS profile active, it will display that for reference. It can display an active terraform workspace, git branch info, python virtual environment and more. Definitely a nice quick productivity booster with no real configuration needed to get going.
{{< gist sheldonhull "93d8060e6f86e0c46535ef6699d6e0c8" "configure-profile-prompt.ps1" >}}
If you are wondering why I didn't leave the pretty awesome "useAcrylic": true on for my main pwsh session, it's because I found the background contrast reduction made it hard to read some darker colors on the prompt.
Be sure to try out the retro pwsh theme for some nice eye candy.
PowerShell Protip: Note the suggested completion based on prior commands in the pwsh prompt. This is some great prerelease work on a better PSReadline experience with Powershell.
I figured I'd go ahead and take this article which I've gutted several times and share some thoughts, even if it's not an authority on the topic. 😀
In the last year, I've been interested in exploring the DevOps philosophy as it applies to operations as well as software development. I originally started various drafts on these concepts back before April 2019, but never got around to finishing it until now.
This isn't a very focused post, more a smattering of thoughts, so let's get to it! 💪
Having come from a development background, applying DevOps to the operational side of the fence is an interesting challenge.
There are so many things that can be unique to the evolving way a team you are part of is growing and learning, that it can be difficult sometimes to know what is "best practice" and was is "essential practice" in a DevOps culture.
There are various ways to implement DevOps, and core principles to DevOps. I've found that one that really resonated with me as I've looked for how others have done this.
Google's Site Reliability Engineering (SRE) material provides a solid guide on the implementation of SRE teams. They consider SRE and DevOps in a similar relationship to how Agile views Scrum. Scrum is an implementation of Agile tenants. Site Reliability Engineering is an implementation of a DevOps culture, with guidelines on how to approach this in a systematic way.
If you think of DevOps like an interface in a programming language, class SRE implements DevOps. Google SRE
What I like about the material, is that a lot of the fixes I've considered to improvements in workflow and planning have been thoughtfully answered in their guide, since it's a specific implementation rather than a broader philosophy with less specific steps.
Regardless of where you are in your journey, a lot of the principles have much to offer. Even smaller organizations can benefit from many of these concepts, with various adjustments being made to account for the capabilities of the business.
Recommended reading if this interests you:
At the beginning of my tech career, I worked at a place that the word "spec" or "requirement" was considered unreasonable. Acceptance criteria would have been looked down upon, as something too formal and wasteful.
While moving towards to implementation of any new project, I was expected to gather the requirements, build the solution, ensure quality and testing, and deploy to production. That "cradle to grave" approach done correctly can promote the DevOps principles, such as ensuring rapid feedback from the end-users and ownership from creation through to production.
I've been in a somewhat interesting blend of roles that gives me some insight into this. As a developer, I always look at things from an 🤖 automation & coding perspective. It's the exception, rather than the norm for me to do it manually without any type of way to reproduce via code.
I've also been part of a team that did some automation for various tasks in a variety of ways, yet often resolved issues via manual interactions due to the time constraints and pressures of inflowing work. Building out integration tests, code unit tests, or any other automated testing was a nice idea in theory, but allocating time to slow down and refactor for automated testing on code, deployments, and other tasks were often deemed too costly or time prohibitive to pursue, in addition to requiring a totally different skillset to focus on.
For example, you can't write code based tests against a deployment, unless you have made the time and effort to learn to code in that language and work through the pain of discovery in writing integration oriented tests. It's not a simple feat 🦶 to just pick up a language and start writing tests in it if you've never done this before.
Reactive work also challenges DevOps focused teams that do operational work.
Google has a lot more detail on the principles of "on-call" rotation work compared to project-oriented work. Life of An On-Call Engineer. Of particular relevance is mention of capping the time that Site Reliability Engineers spend on purely operational work to 50% to ensure the other time is spent building solutions to impact the automation and service reliability in a proactive, rather than reactive manner.
In addition, the challenges of operational reactive work and getting in the zone on solving project work with code can limit the ability to address the toil of continual fixes. Google's SRE Handbook also addresses this in mentioning that you should definitely not mix operational work and project work on the same person at the same time. Instead whoever is on call for that reactive work should focus fully on that, and not try to do project work at the same time. Trying to do both results in frustration and fragmentation in their effort.
This is refreshing, as I known I've felt the pressure of needing to deliver a project, yet feeling that pressure of reactive work with operational issues taking precedence.
It's important to recognize the importance of that operational work. It helps ensure a team is delivering what is most important to a business, a happy customer base! At the same time, always living in the reactive world can easily cause the project related work and reliability improvements that will support those efforts to suffer if a intentional plan to handle this is not made.
I've always been focused on building resilient systems, sometimes to my own detriment velocity wise. Balancing the momentum of delivery features and improving reliability is always a tough issue to tackle. Automation isn't free. It requires effort and time to do correctly. This investment can help scaling up what a team can handle, but requires slower velocity initially to do it right.
How do you balance automating and coding solutions to manual fixes, when you often can't know the future changes in priority?
A leader I've worked with mentioned one great way is to ensure whatever solution you build is immediately applicable to solving work in progress. By this, he meant work that could immediately benefit a team, rather than building things for future possibilities that haven't yet come into play. This aligns with a LEAN mentality and there is a lot to like about it.
Another way I've begun looking at this is solving the key pain-points that are mentioned by others. I tend to look at many improvements, and it can be difficult to choose what area to tackle at times, as I can see value in many of them.
However, solving paint points that are directly impacting someone enough that they are asking for help should take precedence in many cases (not all).
Why? This pain-point solved gains you an immediate win.
If you are solving an issue that others are ok with, or don't recognize as an issue, then you have an uphill battle for adoption compared to solving something immediately impacting their work. This doesn't negate addressing problems that others have not yet recognized, it is just something that's become clear can help in moving forward with improvements with less resistance.
There is an important caveat to all of this. There is a realm of problems when you have expertise in an area that you will identify that others don't see. When assessed against goals of a business, these can be critical.
In my experience, the only way for these identified issues to be clearly prioritized is having a strong ability to communicate the problem and design some clear objectives to place this into the team backlog to address.
Verbally communicating, while important, won't have any staying power compared to a clearly defined objective in a backlog and advocated for in a presentation that helps raise concern for the same issue you care about.
They key is that your signal-to-noise-ratio is good, and when you raise the concern about the issue others don't have to work hard to understand why it's a problem.
SREs balance the juggling of product velocity and reliability improvements in a unique way... the usage of SLO and the error budget concept.
The concepts of having an error budget really struct home for me. Clear Service Level Objectives (SLO) and error budgets to work with helps ensure both the development velocity and stability desired by operations is handled correctly.
Error budgets provide the key ingredient to balancing new feature delivery, while still ensuring happy customers with a stable system.
One of the best sections I've read on this was: Embracing Risk.
Error budgets are discussed, and internal SLI (Server Level Indicators). These are integral to ensuring a balance of engineering effort in balance with new feature delivery. The goal of 100% reliability, while sounding great, is inherently unrealistic.
Product development performance is largely evaluated on product velocity, which creates an incentive to push new code as quickly as possible. Meanwhile, SRE performance is (unsurprisingly) evaluated based upon the reliability of a service, which implies an incentive to push back against a high rate of change. Information asymmetry between the two teams further amplifies this inherent tension.
I've been mulling over this for a while, and find a lot of the concepts very helpful as I've been challenged with both operational and development focus. As always, these are personal thoughts and don't reflect the opinion of my employer. I hope it gives you some food for thought. If you have some feedback, just post up a comment and let's dialogue!
I've been trying to improve modularization with Terraform. Using Terraform Cloud, you get a private module repository. The modules are linked to tags in git.
I've wanted additionally to specifically create releases for the versions in addition to the tags, to make tracking of usable versions a little cleaner.
There are several ways to do this, including using the GitHub API, npm modules, and more. I wanted a simple CLI tool to do this and ran across this great Go utility that worked seamlessly for me.
I've used the Hub cli but the create release syntax never really worked for me.
Github-release is a simple golang cli that worked great for me. Note that this is a fork. This fork is more up to date than the original.
With go installed just run this to get it installed and available in PATH.
To simplify GitHub access, ensure you set an environment variable for your user called GITHUB_TOKEN.
With PowerShell you can do it quickly like this (you might need to close and reopen vscode/terminal for this to be recognized)
To use this, you can chain together some steps and see how it can save you time on creating a GitHub release.
{{< gist sheldonhull "53055bbff368a4ebe4e0794076a56c37" >}}
This helped me get moving faster ⚡ on using Github releases without the tedious work to create. If this helped you or have any feedback, drop a comment below and let me know! The comments are powered by Utterances which will open a Github issue to discuss further. 👍
Documentation is such an important part of a developer's life. I think we often take it for granted, and it's an afterthought in many projects. However, as I consider my work, I know that I'm not reinventing the wheel very often 😀. Most of what I do is built on the back of others' work. When I use tooling, I'm reading the documentation and using it as my basis to get work done. When I use my notes and blog posts as a reference, I'm using my informal version of knowledge gathering.
INVEST in documenting your work as you go, for the person behind you. You don't find time to do it, you make time to do it while you work, as a first class citizen of your work, not an after-thought. Think of all the times you've had to dig for answers and save someone else that experience.
You code and document not as much for yourself, but for the person that comes behind you.
I've found a happy solution in the Asciidoctor documentation format over markdown. You can go google this for more expanded understanding, but I've decided that other than for basic notes and blog posts which are very simplistic, I now choose Asciidoctor.
Why use Asciidoc format over markdown comes down to the needs of technical documentation.
Here are some key reasons why I've found Asciidoc format to be worth learning:
include::file[] statement, while markdown would require me to embed my code directly as a code block.IMPORTANT: fooSince the common documentation system used where I am at is Confluence, I decided to leverage the incredible confluence-publisher project that made this entire process a breeze. Check the repo and the linked documentation out here: Confluence Publisher
In the future, if I didn't use confluence, I'd explore rendering as a static website through Hugo (that's what this site is generated from) or revisit Antora and maybe merge my content into the format required by Atora programmatically.
Since Asciidoc is written in Ruby, use docker and you won't have to deal with dependency nightmares, especially on Windows.
$RepoDirectoryName = 'taco-ops-docs'
echo "🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮"
echo "Running confluence publisher 🌮"
echo "📃 Publishing $RepoDirectoryName repo contents"
docker run --rm -v $BUILD_SOURCESDIRECTORY/$RepoDirectoryName/docs:/var/asciidoc-root-folder -e ROOT_CONFLUENCE_URL=$ROOT_CONFLUENCE_URL \
-e SKIP_SSL_VERIFICATION=false \
-e USERNAME=$USERNAME \
-e PASSWORD=$PASSWORD \
-e SPACE_KEY=$SPACE_KEY \
-e ANCESTOR_ID=$ANCESTOR_ID \
-e PUBLISHING_STRATEGY=$PUBLISHING_STRATEGY \
confluencepublisher/confluence-publisher:0.0.0-SNAPSHOT
echo "📃 Publishing $RepoDirectoryName repo contents finished"
Yes... I know. I get bored reading log messages when debugging so my new year premise was to add some emoji for variety. Don't judge. 😁
So the above approach is fantastic for a single repo. I wanted to take it to a different level by solving this problem for distributed documentation. By distributed I meant that instead of containing all the documentation in a single "wiki" style repo, I wanted to grab documentation from the repositories I choose and render it. This would allow the documentation related to being contained in the repository it is related to.
For instance, what if I wanted to render the documentation in the following structure:
** General Documentation**
taco-ops-runbook
---> building-tacos
--------> topic.adoc
---> eating-tacos
--------> topic.adoc
---> taco-policies
--------> topic.adoc
---> taco-as-code
--------> topic.adoc
** Repo Oriented Documentation**
github-repos
---> taco-migration
--------> category-1
------------> topic.adoc
------------> topic.adoc
--------> category-2
------------> topic.adoc
------------> topic.adoc
---> taco-monitoring
--------> category-1
------------> topic.adoc
------------> topic.adoc
--------> category-2
------------> topic.adoc
------------> topic.adoc
The only current solution found was Antora.
Antora is very promising and great for more disciplined documentation approaches by software development teams.
The limitation I faced was complexity and rigidity in structure.
For Antora to generate a beautiful documentation site, you have to ensure the documentation is structured in a much more complex format.
For example, the docs might be under docs/modules/ROOT/pages/doc.adoc and have a nav.adoc file as well.
While this promises a solid solution, retrofitting or expecting adoption might be tricky if your team has never even done markdown.
I ended using an Azure DevOps pipeline (YAML of course 🤘) that provides a nice easy way to get this done.
First, for proper linking, you should follow the directions Azure DevOps gives on the creation of a Github Service Connection which uses OAUTH. This will ensure your setup isn't brittle and using your access token.
{{< gist sheldonhull "053cb176d5c2847a4e323f01207acb82" >}}
** Repo Oriented Documentation**
taco-ops-repo
README.adoc -- optional, but I always include for landing page, and point to the docs folder using link:./docs/myrepo.adoc
---> [docs]
------> [resources] -- optional, but keeps the scripts organized and consistent, or any images
------> process.adoc
------> another-process.adoc
---> taco-ops-repo.adoc
include::./resources/_myscript.ps1[]. You may have to test that relative path issue if doing multiple repos.{{< admonition type="info" title="Docker Commands in Microsoft-Hosted Agent" >}} I didn't expect docker commands to work in Azure DevOps agent, thinking nested virtualization would not work and all. However, it works beautifully. Consider using Azure DevOps yaml pipelines for running your docker commands and you take one step towards better build processes. {{< /admonition >}}
{{< admonition type="info" title="Updated 2020-05-05" >}}
I've had lots of challenges in getting docker for sql-server working because I've wanted to ensure for my dev use case that there was no need for virtual volume management and copying files into and out of this. Instead, I've wanted to bind to local windows paths and have it drop all the mdf/ldf right there, so even on container destruction everything is good to go.
After working through the changes in SQL 2019 that require running as non-root, I've gotten it work again. No install of sql-server needed. Easy disposable development instance through docker! I'll update my docker compose content when I can, but in the meantime, this should get you running even more quickly with SQL Server 2019.
docker run `
--name SQL19 `
-p 1433:1433 `
-e "ACCEPT_EULA=Y" `
-e "MSSQL_SA_PASSWORD=ThisIsNotARealPassword@!1}" `
-v C:\mssql\SQL19:/sql `
-d mcr.microsoft.com/mssql/server:2019-latest
docker run `
--name SQL19WithSpaces `
-p 1434:1433 `
-e "ACCEPT_EULA=Y" `
-e "MSSQL_SA_PASSWORD=ThisIsNotARealPassword@!1}" `
-v C:\mssql\SQL19WithSpaces:/sql `
-d mcr.microsoft.com/mssql/server:2019-latest
{{< /admonition >}}
Say you have an onboarding and need to get a new developer up to speed. Have you gone through the setup process for SQL Server on Windows? It's a bit tedious and even with running scripts to install (or in my case I wrapped up with Chocolatey), it's still a lot of things you need, including possibly a reboot if missing some .net dependencies.
The normal solution is to leverage more localdb for quickly deploying and isolating the database.
This has its own set of challenges that personally I like to avoid. Localdb is more isolated, but it's not truly like running a local SQL Server standard edition, and debugging permissions, sharing, and other things can with its own set of steps to work through. I've seen it common for many devs to just avoid these issues and run a local SQL Server installation to just simplify and work with SQL Server as it's expected to be.
I'd explored Docker SQL Server containers in the past, but one big issue for adoption to me was the issues I had mounting the local Windows folders to the Linux based container. ❓ Why is this important? Ease of usage for a developer. If I proposed this would save effort to many developers working with SQL Server, I'm pretty sure telling them that they'd have to copy a backup or database file in via CLI to a virtual mounted drive that they can't easily manage would be a deal-breaker. I basically wanted to ensure if they spun up SQL Server in a container, the database backups, mdf and ldf files, and any other created files would be able to persist outside of that instance, ensuring easy development workflow.
I'm happy to say I finally have it working to my satisfaction, resolving most of those concerns.
If you've desired to do the following, then using Docker might end up saving you some effort.
This is not for production. There's more effort and debate that goes on to using SQL Server in containers, Kubernetes (even more complex!), and other containerization approaches that are far outside the scope of what I'm looking at. This is first and foremost focused on ensuring a development workflow that reduces complexity for a developer and increases the ease at which SQL Server testing can be implemented by making the SQL Server instance easily disposed of and recreated.
If this also means I spread some 🐧 Linux magic to some Windows stack developers... then count it as a sneaky win. 😏
The following example is done with SQL Server on Linux. As I'm already comfortable with SQL Server on Windows, I wanted to be try this on Linux based container. I also wanted to continue using Linux based containers for tooling, and not have to switch back to Windows containers for the sole purpose of running SQL Server. At the time I began this testing, I found it was exclusive. You either ran Linux or Windows-based containers. This is changing with the advent of new features in Docker that are there to allow side by side Windows + Linux based containers.
Release notes indicate:
Experimental feature: LCOW containers can now be run next to Windows containers (on Windows RS3 build 16299 and later). Use --platform=linux in Windows container mode to run Linux Containers On Windows. Note that LCOW is experimental; it requires the daemon experimental option. Docker Community Edition 18.03.0-ce-win59 2018-03-26
The main difference in your local development process will be Windows Authentication vs SQL Authentication. Use SQL Authentication with Linux based SQL Server
The following is the result of a lot of trial and error over a year. Thanks to Shawn Melton 👏 also for providing me with support doing my troubleshooting as part of the SQL Server community in Slack. You can find Shawn's example I used as a starting point for my 🧪testing in this gist.
A few helpful tips:
-d for detached and you can see the SQL Server console output in the console.{{< gist sheldonhull a70a3a731b329b67f47a331c64c72ab5 >}}
Another reason I'm really excited about is the ability to better support testing through tools like Pester. Yes, it's a geek thing, but I love a solid written test 🧪that maintains my code and helps me ensure quality with changes. Better TDD is a goal of mine for sure.
🔨 This supports implementation of better tests by providing the potential for spinning up a local SQL Instance, restoring a test database, and running a sequence of actions against it with pass or fail without the additional infrastructure requirements to have this done on another server. Making your tests that are not full integration testing as minimally dependent on external factors is a fantastic step to saving you a lot of work.
A simple pester (PowerShell) might frame the start of a test like this:
Before All {
docker-compose up -d
Import-Module Dbatools
# Wait Until dbatools confirms connectivity through something like test-dbaconnection, then proceed with tests
# Test-DbaConnection ....
# Restore-DbaDatabase ...
}
After All {
docker-compose down --volume
}
Describe "DescribeName" {
Context "ContextName" {
It "ItName" {
Assertion
}
}
}
Hope this helps someone. I spent at least a year coming back over time to this hoping to actually get it working in a way that felt like a first-class citizen and reduced complexity for development work.
I'm pretty happy with the results. 😁