Skip to content

posts🔗

  • 3 min read

Go R1 Day 61

progress

It was a dark and stormy night. The world was bleak. A command was sent to The Compiler.

The assumptions were flawed. The command was rejected.

You have given me an invalid command with -argument-that-shall-not-work 5 being invalid.

But I've provided --argument-that-shall-not-work, the indomitable wizard said.

Your command is unworthy.

Digging into esoteric tomes of knowledge, the wizard discovered others have had similar issues when calling external processes using the legendary wizardry of os/exec. However, none could shine light into the darkness of his failure.

Running in pwsh worked fine.

Next, the wizard tried a variety of escaping commands.

  • Using shellescape package.
  • Using back-ticks with the arguments to escape.
  • Using partially quoted arguments in the slice of the strings.
  • Using no quotes.
  • Went down the path of ancient texts describing similar issues.1

To the wizards dismay, copying the printed debug output worked fine in the terminal, but alas would not be executed by The Compiler.

It began to feel like the curse of dynamic SQL queries that had long plagued the wizard until PowerShell had been discovered.

The wizard ruminated on his plight. He thought:

At the end of the day, all things seem to come down to strings and the cursed interpretation of my textual commands to The Compiler. How many a day have I wasteth upon the fault of a single character. The root of all evil must be a string."

The wizard connected to a new remote instance, using the power of the Remote SSH plugin and began debugging in VSCode.

The debug breakpoint config that worked was set in stone.

  {
      "name": "Run frustrating-test",
      "type": "go",
      "request": "launch",
      "mode": "debug",
      "program": "${workspaceFolder}/MyTestApp/test.go",
      "args": [
        "-count",
        "100",
        "-batch",
        "10",
        "-delay",
        "1",
        "-server",
        "MyCustomIPAddress",
        "-debug",
      ],
      "debugAdapter": "legacy" // newer version wouldn't work remote
    },

Consulting The Sage (aka Debugger), it advised the wizard of the message sent to The Compiler.

Debug Variables

The wizard had a revelation. A fellow wizard advised to break the appending into individual statements instead of trying to do so much in one step.

The incantation changed from:

command = append(command, fmt.Sprintf(`--header "user-id: %s"`, petname.Generate(petNameLength, "-"))

to the following:

command = append(command, "--header")
command = append(command, fmt.Sprintf(`user-id: %s`, petname.Generate(petNameLength, "-")))
command = append(command, "--max-retry-count", "5")

The foe vanquished, the The Blight of Strings was cast aside with malice. The wizard swore to never fall prey to this again.

{{< admonition type="Note" title="Further Detail (:(fas fa-external-link-alt): expand to read)" open=false >}}

Josesh1 pointed towards: EscapeArgs. I did not find any equivalent for Darwin. The closest thing I could find was execveDarwin which I believe is the execution line, which gets the argument list from: SlicePtrFromStrings which is defined at here

I'll have to re-examine in the future when I have more experience with Go, as it's not a simple chain to follow. :(fas fa-brain):

[test --param 1]
strings.Join(a)... "test --param 1"
os.Command: ["test" "--param 1"]
echo "test --param 1\n"

This pointed towards a similar issue with the \n showing up.

:(fas fa-play): Playground

{{< /admonition >}}

The Compiler's heartless gaze felt nothing. In the shadows, The String Balrock bid its time, knowing that the wizard would suffer once again.

It Works

  1. 2016 // Prevent Escaping exec.Command Arguments in Go | Joseph Spurrier ↩↩

  • 1 min read

Go R1 Day 60

progress

  • Finished up the basics of dependency injection and how this helps with testable code.
  • Worked through concurrency test using channels and go routines. This will take a bit more to get comfortable with as there is a variety of concepts here. My initial attempts finally started working using an anonymous function, but couldn't finalize due to some issue with launching the parallel executable being called. Not sure why the --argname seemed to be trimming the first dash from the argument when using args = append(args,"--argname 5").

I sure enjoy the visuals from pterm library. When not using the -debug flag, the concurrent counter reported a nice increase of total threads and then exited upon failure of any goroutine.

  • 1 min read

Go R1 Day 59

progress

  • Built some Mage tasks for setup of local projects.
  • Used retool post from Nate Finch with Mage project and found it promising to vendor Go based tooling into a local project and not rely on the global package library.
  • Created magetools repo to house some basic general mage tasks I could share with other projects.
year, month, day := time.Now().Date()
dateString := fmt.Sprintf("%d-%02d-%d", year, month, day)

Use padding in the Sprintf call to ensure a date comes back with 07 instead of 7.

  • 3 min read

Go R1 Day 57

progress

  • Did some adhoc work in this repo (the hugo repo that contains this blog) testing out Mage, which is a Go based Make alternative.
  • Generated dynamic target directory for hugo posts using stringify for kebab case.
  • Unexpected behavior when generating dynamic file path including date.
year, month, day := time.Now().Date()
str := stringy.New(title)
slugTitle := strings.Join([]string{string(year), string(month), string(day), str.KebabCase("?", "").ToLower()}, "-")

The output failed to handle the year, resulting in some odd \x18 path generation.

In reviewing the values from returned from time.Now().Date, I realized it wasn't an int value being returned.

To work through the definition, I figured this would be a good chance to use the cli only to find the docs.

go doc 'time.Now'

returned the following:

package time // import "time"

func Now() Time
    Now returns the current local time.

To get the source code for the function:

  • go doc -src 'time.Now'
  • go doc -src 'time.Date'

This shows the return value of Date() is actually time type, not int.

Still couldn't see where the multiple return parameters were defined so I ran:

go install golang.org/x/tools/cmd/godoc@latest
godoc --http 127.0.0.1:3030

Ok... Figured it out. I was looking at the func Date(). However, what I should have been looking at was the exported method func (Time) Date. This correctly shows:

func (t Time) Date() (year int, month Month, day int)

I still couldn't figure this out until I tried running it in the playground.

:(fa-fas fa-play): Playground - Demonstrate Problem

./prog.go:11:37: conversion from int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)

Runes. Strings. I know folks say it boils down to 1's and 0's, but dang it... seems like my life always boils down to strings. 😆

:(fa-fas fa-play): Playground - Fixed The Problem

Finally got it all working.

Strongly typed languages are awesome, but this type of behavior is not as easy to figure out coming from a background with dynamic languages. I

In contrast, PowerShell would be: Get-Date -Format 'yyyy'.

Here's an example of the Mage command then to generate a blog post with a nice selection and prompt.

// calculatePostDir calculates the post directory based on the post title and the date.
func calculatePostDir(title string) string {
    year, month, day := time.Now().Date()
    str := stringy.New(title)
    kebabTitle := str.KebabCase().ToLower()
    slugTitle := strings.Join(string(year), string(month), string(day),kebabTitle, "-") ///stringy.ToKebabCase(title)

    pterm.Success.Printf("Slugify Title: %s", slugTitle)
    filepath := filepath.Join(contentDir, string(year), slugTitle)
    pterm.Success.Printf("calculatePostDir: %s", slugTitle)
    return filepath
}

Then creation of the post:

// New namespace groups the new post generatation commands.
type New mg.Namespace
// NewPost creates a new post in the Hugo format.
func (New) Post() error {
    prompt := promptui.Select{
        Label: "Select Type of Post j/k to navigate",
        Items: []string{"100DaysOfCode", "microblog", "blog"},
    }
    _, result, err := prompt.Run()
    if err != nil {
        pterm.Success.Printf("Prompt failed %v\n", err)
        return err
    }
    pterm.Success.Printf("New Post: [%s]", result)
    promptTitle := promptui.Prompt{
        Label: "Enter Title",
    }
    title, err := promptTitle.Run()
    if err != nil {
        pterm.Error.Printf("Prompt failed %v\n", err)
        return err
    }
    // the archetype in archtytpes directory to use
    var kind string

    switch result {
    case "100DaysOfCode":
        kind = "code"
    default:
        kind = result
    }
    fileName := calculatePostDir(title)
    if err := sh.RunV("hugo", "new", fileName, "--kind", kind); err != nil {
        return err
    }
    return nil
}
  • 1 min read

Go R1 Day 58

progress

  • Avoiding a panic in Go for missing dictionary match is very straight forward. The error pattern for failed conversions and out of range index matches is the same, with: ok, err := action.
  • TODO: Figure out if ok to reference an error in a test by: is.Equal(error.Error(),"unable to find value in map"). Linter warns me with: Method call 'err.Error()' might lead to a nil pointer dereference.
  • Started work with dependency injection.
  • 2 min read

Go R1 Day 55

progress

In Go, when you call a function or a method the arguments are copied. 1

  • Built some test cases for working with pointers and test methods.
  • Did this in Goland to become more familar with it and figured out how to use the nice tests explorer with a floating window to auto-test.
  • Built a filewatcher configuration watcher as alternative to tests panel, allowing automated run in terminal or output panel of go test ./....
  • Couldn't figure out how to run every test in project, as it's a multi-module repo (for tutorials). VSCode works fine with this with the multi-module support.

These pointers to structs even have their own name: struct pointers and they are automatically dereferenced. 1

This explains something that felt a little confusing as a new Gopher. Now I know why returning a value back as an int didn't require explicit dereferencing. 2

func (w *Wallet) Balance() int {
  return w.balance // <---- automatically deferenced
}

Not clear yet on if I need to set the is := New(t) in the context of each t.Run() or not.

t.Run("withdraw 2 amounts", func(t *testing.T) {
        wallet := pointers.Wallet{}
        wallet.Deposit(pointers.Bitcoin(20))
        err := wallet.Withdraw(pointers.Bitcoin(5))
        is.NoErr(err) // Withdraw should have no errors
        assertBalance(t, &wallet, pointers.Bitcoin(15))
    })

  1. Pointers ↩↩

  2. Method Values ↩

  • 3 min read

My Goland Log

Purpose

This is a log of my second journey in using Goland from Jetbrains for Go development.

I've got VSCode in a great state. It's flexible, powerful, and I've highly customized it to my workflow.

However, doing Go development, I'd like to better explore Jetbrains Goland and see if the experience proves positive.

I'll log updates and issues here as I work through it in the hope that it might provide you better information if you are considering Goland as well.

Emoji Definition
🔴 Not solved yet
🟠 Issue or behavior unclear
🚩 It's an issue I believe I understand but don't like the answer to
✅ Understand and no issues with behavior

Goland Log

{{< admonition type="Question" title=":(fas fa-vials): Testing" open=false >}}

🔴 Not certain yet how to run multi-module repo based tests through project (VSCode supports with experimental flag).

✅ Run individual directory test. This also generates a run configuration (ephemeral) that can be persisted as a test configuration for the project.

test output

test coverage on file list

✅ Toggle auto-run for testing is possible, so upon save, it will run after a 1 second delay. The built in test renderer is better than VSCode's Go test output. Not only does it render the test with color1 but also organizes into a nice test explorer view.

Overall, the test output is polished, able to be undocked and run in a separate window. Works well for a TDD workflow.

Goland Test Coverage

@s0xzwasd provided some great insight on the different approach in the comments on this page. Compound configurations work well to run multiple sets of test when using multiple modules in the same repo. I tried this, and while tedious to click through the first time, It's easier to configure from a UI standpoint than trying to work through the VSCode tasks.json schema and build tasks by hand.

Goland Compound Test Coverage Explorer Output

{{< /admonition >}}

{{< admonition type="Question" title=":(fas fa-running): Run Configurations" open=false >}}

🔴 Not able to set dynamic $FileDir$ in the run configurations it seems.

🟠 Project configuration resolves path to relative once saved, but displays with absolute. This is confusing.

✅ Can include configuration in project so anyone that opens in Goland can run the same prebuilt tasks.

{{< /admonition >}}

{{< admonition type="Question" title=":(fas fa-search): Linting" open=false >}}

🟠 Working on golangci-lint integration. There is an extension and I've also configured the FileWatcher to run on save, but neither is working as seamlessly as VSCode setting as the gopls linter tool.

{{< /admonition >}}

{{< admonition type="Question" title=":(fas fas fa-tools): Refactoring & Fixes" open=false >}}

🟠 Can't find a way to assign intentions to a keyboard shortcut. For example, Add a Go Comment header requires a click on a lightbulb icon, and can't find a way to allow this to be triggered by a keyboard shortcut.

{{< /admonition >}}

  1. vscode is requires -nocolor type argument to avoid console escaping if using a library that uses color. ↩

  • 1 min read

Go R1 Day 54

progress

  • worked with tests and Goland.
  • Modified table driven tests to remove hard coded test case inputs.
  • 4 min read

Steampipe Is Sql Magic

Up And Running In Minutes

I tried Steampipe out for the first time today.

:(fab fa-twitter fa-fw): Follow Steampipe On Twitter

I'm seriously impressed.

I built a project go-aws-ami-metrics last year to test out some Go that would iterate through instances and AMIs to build out aging information on the instances.

I used it to help me work through how to use the AWS SDK to iterate through regions, instances, images, and more.

In 15 mins I just solved the equivalent issue in a way that would benefit anyone on a team. My inner skeptic was cynical, thinking this abstraction would be problematic and I'd be better served by just sticking with the raw power of the SDK.

It turns out this tool already is built on the SDK using the same underlying API calls I'd be writing from scratch.

First example: DescribeImage

This is the magic happening in the code.

    resp, err := svc.DescribeImages(&ec2.DescribeImagesInput{
        Owners: []*string{aws.String("self")},
    })
    for _, image := range resp.Images {
        d.StreamListItem(ctx, image)
    }

This is the same SDK I used, but instead of having to build out all the calls, there is a huge library of data already returned.

    req, publicImages := client.DescribeImagesRequest(&ec2.DescribeImagesInput{
        Filters: []*ec2.Filter{
            {
                Name:   aws.String("is-public"),
                Values: []*string{aws.String("true")},
            },
        },
    },
    )

There is no need to reinvent the wheel. Instead of iterating through regions, accounts, and more, Steampipe allows this in plain old SQL.

Query The Cloud For example, to gather:

  • EC2 Instances
  • that use AWS Owned Images
  • and use an image that created greater than n period
  • and want the aging in days
SELECT
    ec2.instance_id,
    ami.name,
    ami.image_id,
    ami.state,
    ami.image_location,
    ami.creation_date,
    extract('day' FROM now()) - extract('day' FROM ami.creation_date) AS creation_age,
    ami.public,
    ami.root_device_name
FROM
    aws_ec2_ami_shared AS ami
    INNER JOIN aws_ec2_instance AS ec2 ON ami.image_id = ec2.image_id
WHERE
    ami.owner_id = '137112412989'
  AND ami.creation_date > now() - INTERVAL '4 week'

There are plugins for GitHub, Azure, AWS, and many more.

You can even do cross-provider calls.

Imagine wanting to query a tagged instance and pulling the tag of the work item that approved this release. Join this data with Jira, find all associated users involved with the original request, and you now have an idea of the possibility of cross-provider data Steampipe could simplify.

Stiching this together is complicated. It would involve at least 2 SDK's and their unique implementation.

I feel this is like Terraform for Cloud metadata, a way to provide a consistent experience with syntax that is comfortable to many, without the need to deal with provider specific quirks.

Query In Editor

  • I downloaded the recommended TablePlus with brew install tableplus.
  • Ran steampipe service start in my terminal.
  • Copied the Postgres connection string from the terminal output and pasted into TablePlus.
  • Pasted my query, ran, and results were right there as if I was connected to a database.

TablePlus

AWS Already Has This

AWS has lots of ways to get data. AWS Config can aggregate across multiple accounts, SSM can do inventory, and other tools can do much of this.

AWS isn't easy. Doing it right is hard. Security is hard.

Expertise in building all of this and consuming can be challenging.

🎉 Mission accomplished!

Experience

I think Steampipe is offering a fantastic way to get valuable information out of AWS, Azure, GitHub, and more with a common language that's probably the single most universal development language in existenance: SQL.

One of the goals of Steampipe since we first started envisioning it is that it should be simple to install and use - you should not need to spend hours downloading pre-requisites, fiddling with config files, setting up credentials, or pouring over documentation. We've tried very hard to bring that vision to reality, and hope that it is reflected in Steampipe as well as our plugins.

Providing a cli with features like this is incredible.

  • execute
  • turn into an interactive terminal
  • provide prompt completion to commands
  • a background service to allow connection via IDE

The Future

The future is bright as long as truncate ec2_instance doesn't become a thing. 😀

Further Resources

If you want to explore the available schema, check out the thorough docs.