Skip to content

Go R1 Day 70

progress🔗

  • Concurrency section wrap-up with Learn Go With Tests.
  • Reviewed material learned from: Go R1 Day 61
  • Read material, but didn't do a lot of tests on this, per mostly concept oriented. Used concurrent progressbar example from uiprogress project to test concurrent UI updates.
  • My last concurrency test case was to launch many concurrent processes for a load test. This didn't leverage goroutinues as typically used, since it was calling to an executable on the host machine. However, this provided a great use case for something I've done before with DevOps oriented work and showed how to use concurrency as a blocking operation. Once the user was done with the test, ctrl+c was used to kill the active requests and the program exited.
  • I need more practice with channels. I was only wanting error stdout content, and so didn't have any need for receiving channel output back in a structured way. This is probably an atypical usage of concurrency, fitting for an external load test, but not internal Go code.
  • Still found it pretty cool that I could spin up 500 processes at once, with far less overhead than doing in PowerShell.

{{< admonition type="Note" title="Example Of Doing In PowerShell" open=true >}} Doing this in PowerShell is far more compact, but not as performant.

This is a good example of the different in using Go for adhoc tasks. It will require more code, error handling care, but pays off in something that is likely more stable and easier to run across multiple systems with a single binary.

#!/usr/bin/env pwsh
$Server = 'IPADDRESS'
$ServerPort = '3000'
Write-Host 'Load Test Start'
$RandomPort = '4000'

$j = @(4000..4100)| ForEach-Object {
    $c = $_
    Start-ThreadJob -ThrottleLimit 1000 -StreamingHost $Host -InputObject $c -ScriptBlock {
    $RandomPort = $input
    &mybinary serve --max-retry-count 5 --header "user-id: $(petname)" --header "session-id: $(uuidgen)" "${using:Server}:${using:ServerPort}"
}
}
$j | Wait-Job | Receive-Job
$j | Stop-Job

I didn't benchmark total load difference between this and Go, but I'm sure the pwsh threads were a bit more costly, though for this test case might not have been a large enough value to make much difference.

{{< /admonition >}}

Code Examples🔗

This first section is the startup. Key points:

  • main() is the entry point for the program, but doesn't contain the main logic flow. Inspired by Matt Ryer's posts, I now try to ensure main is as minimal as possible to encourage easier automation in testing. Since run contains the main logic flow, the actual CLI itself can be called via integration test by flipping to Run() and calling from testing file using a blackbox testing approach.
package main

import (
    "bytes"
    "errors"
    "flag"
    "fmt"
    "io"
    "math"
    "os"
    "os/exec"
    "strings"
    "sync"
    "time"

    shellescape "github.com/alessio/shellescape"
    petname "github.com/dustinkirkland/golang-petname"
    "github.com/google/uuid"
    "github.com/pterm/pterm"
    "github.com/rs/zerolog"
    "github.com/rs/zerolog/log"
)

const (
    // exitFail is the exit code if the program
    // fails.
    exitFail = 1

    // desiredPort is the port that the app forwards traffic to.
    desiredPort = 22

    // petnameLength is the length of the petname in words to generate.
    petNameLength = 2

    // startingPort is the starting port for a new connection, and will increment up from there so each connection is unique.
    startingPort = 4000

    // maxRetryCount is the number of times to retry a connection.
    maxRetryCount = 5
)

func main() {
    if err := run(os.Args, os.Stdout); err != nil {
        fmt.Fprintf(os.Stderr, "%s\n", err)
        os.Exit(exitFail)
    }
}

Next run contains the main logic flow. The goal is that all main program logic for exiting and terminating is handled in this single location.

// Run handles the arguments being passed in from main, and allows us to run tests against the loading of the code much more easily than embedding all the startup logic in main().
// This is based on Matt Ryers post: https://pace.dev/blog/2020/02/12/why-you-shouldnt-use-func-main-in-golang-by-mat-ryer.html
func run(args []string, stdout io.Writer) error {
    if len(args) == 0 {
        return errors.New("no arguments")
    }
    InitLogger()
    zerolog.SetGlobalLevel(zerolog.InfoLevel)

    debug := flag.Bool("debug", false, "sets log level to debug")
    Count := flag.Int("count", 0, "number of processes to open")
    delaySec := flag.Int("delay", 0, "delay between process creation. Default is 0")
    batchSize := flag.Int("batch", 0, "number of processes to create in each batch. Default is 0 to create all at once")
    Server := flag.String("server", "", "server IP address")
    ServerPort := flag.Int("port", 3000, "server port") //nolint:gomnd

    flag.Parse()
    log.Logger.Info().Int("Count", *Count).
        Int("delaySec", *delaySec).
        Int("batchSize", *batchSize).
        Str("Server", *Server).
        Msg("input parsed")

    log.Logger.Info().
        Int("desiredPort", desiredPort).
        Int("petNameLength", petNameLength).
        Int("startingPort", startingPort).
        Msg("default constants")

    if *debug {
        zerolog.SetGlobalLevel(zerolog.DebugLevel)
    }

    RunTest(*Count, *delaySec, *batchSize, *Server, *ServerPort)
    return nil
}

Next, InitLogger is used to initialize the logger for zerolog. I don't need multiple configurations right now so this is just stdout.

// InitLogger sets up the logger magic
// By default this is only configured to do pretty console output.
// JSON structured logs are also possible, but not in my default template layout at this time.
func InitLogger() {
    output := zerolog.ConsoleWriter{Out: os.Stdout, TimeFormat: time.RFC3339}
    log.Logger = log.With().Caller().Logger().Output(zerolog.ConsoleWriter{Out: os.Stderr})

    output.FormatLevel = func(i interface{}) string {
        return strings.ToUpper(fmt.Sprintf("| %-6s|", i))
    }
    output.FormatMessage = func(i interface{}) string {
        return fmt.Sprintf("%s", i)
    }
    output.FormatFieldName = func(i interface{}) string {
        return fmt.Sprintf("%s:", i)
    }
    output.FormatFieldValue = func(i interface{}) string {
        return strings.ToUpper(fmt.Sprintf("%s", i))
    }
    log.Info().Msg("logger initialized")
}

Test the existence of the binary being run in a load test, and exit if it doesn't exist. This should more likely be handled in the run fuction, but I just did it here for simplicity in this adhoc tool.

// TestBinaryExists checks to see if the binary is found in PATH and exits with failure if can't find it.
func TestBinaryExists(binary string) string {
    p, err := exec.LookPath(binary)
    if err != nil {
        log.Logger.Error().Err(err).Str("binary",binary).Msg("binary not found")
        os.Exit(exitFail)
    }

    return p
}

Next, buildCLIArgs handles the argument string slice construction. I learned from this to keep each line and argument independent as escaping has some strange behavior if you try to combine too much in a single statement, especially with spaces. Best practice is to keep this very simple.

// buildCliArgs is an example function of building arguments via string slices
func buildCliArgs(Server string, ServerPort int, port int) (command []string) {
    command = append(command, "server")
    command = append(command, "--header")
    command = append(command, fmt.Sprintf(`user-id: %s`, petname.Generate(petNameLength, "-")))
    command = append(command, "--header")
    command = append(command, fmt.Sprintf(`session-id: %s`, uuid.Must(uuid.NewRandom()).String()))
    command = append(command, "--max-retry-count", "5")
    command = append(command, Server+":"+fmt.Sprintf("%d", ServerPort))
    return command
}

Finally, a function that run the tests with some pretty output using pterm. This would be probably better to break-up for testing, but again, adhoc project, so this ended up working decently as I was learning concurrency.

// RunTest is the main test function that calculates the batch size and then launches the  creation using a routinue.
func RunTest(Count int, delaySec int, batchSize int, Server string, ServerPort int) {
    log.Logger.Info().Msg("RunTest startings")
    totalBatches := math.Ceil(float64(Count) / float64(batchSize))
    log.Logger.Info().Float64("totalBatches", totalBatches).Msg("batches to run")
    myBinary := TestBinaryExists("binaryname")
    port := startingPort
    var wg sync.WaitGroup

    totals := 0
    p, _ := pterm.DefaultProgressbar.WithTotal(Count).WithTitle("run s").Start()

    for i := 0; i < int(totalBatches); i++ {
        log.Debug().Int("i", i).Int("port", port).Msg("batch number")

        for j := 0; j < batchSize; j++ {
            if totals == Count {
                log.Debug().Msg("totals == Count, breaking out of loop")

                break
            }

            totals++
            log.Debug().Int("i", i).Int("", totals).Msg("")
            cmdargs := buildCliArgs(Server, ServerPort, port)
            wg.Add(1)
            go func() {
                defer wg.Done()
                buf := &bytes.Buffer{}
                cmd := exec.Command(, cmdargs...)
                cmd.Stdout = buf
                cmd.Stderr = buf
                if err := cmd.Run(); err != nil {
                    log.Logger.Error().Err(err).Bytes("output", buf.Bytes()).Msg(" failed")
                    os.Exit(exitFail)
                }
                log.Logger.Debug().Msgf(" %v", shellescape.QuoteCommand(cmdargs))
                log.Logger.Debug().Bytes("output", buf.Bytes()).Msg("")
            }()

            p.Title = "port: " + fmt.Sprintf("%d", port)
            p.Increment()
            port++
        }
        time.Sleep(time.Second * time.Duration(delaySec))
    }
    p.Title = "s finished"
    _, _ = p.Stop()
    wg.Wait()
}