A standard way of running shell commands in Go is via exec.Command.
However, calling this function directly within your business logic makes the code hard to test.
This post demonstrates an approach I developed when I had to adapt my code for running shell commands both locally and remote over SSH, while also keeping it testable.
TL;DR: extract exec.Command into an interface and create a mock implementation for testing. Implement it for your SSH client too, if needed. See example code here.
The code below assumes it is executed in a unix-like operating system where sh shell is available. This simplifies the whole enterprise quite a bit, but also brings some limitations.
A Shell interface and implementation
Let’s extract exec.Command function into a simple interface:
type Shell interface {
Execute(ctx context.Context, cmd string) (output []byte, err error)
}
It is very self-explanatory. Context is used to cancel long-running commands (a standard library allows doing so, which is cool!).
Now implement this interface with a standard library:
type LocalShell struct{}
func (_ LocalShell) Execute(ctx context.Context, cmd string) ([]byte, error) {
wrapperCmd := exec.CommandContext(ctx, "sh", "-c", cmd)
return wrapperCmd.CombinedOutput()
}
The implementation is simple as well, except you might be wondering why is it wrapped into sh -c?
This is because exec.Command is a little more advanced than we need it to be right now. It treats cmd argument as a program name and tries to resolve its absolute path, which we’d like to avoid. It also doesn’t allow using pipes and output redirection in a command string easily, so e.g. exec.Command("echo hello > greeting.txt") wouldn’t work as expected. By wrapping a command with sh -c we force Go to execute the command as is.
If our OS can run ls, we can test this function:
func TestLocalShell_Execute_ls(t *testing.T) {
shell := LocalShell{}
output, err := shell.Execute(context.Background(), "ls")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(output), "local_shell_test.go") {
t.Errorf("expected output to contain a current go file, got:\n%s", string(output))
}
}
Beware, this is not a unit-test that we’re here for. The command being executed here is not under our control. While ls is a simple, quick and predictable command, this approach is not really viable for calls to more complex programs. For example, how could we test commands that modify external state (rm -rf, anyone?), or commands that run for several hours, like database backups? This is where unit tests come to the rescue.
Unit-testing shell commands
Now let’s assume we have to launch another 3rd-party program from our Go process, whose behavior isn’t easily predictable or testable.
Moreover, that program may not even be available on a development machine. With Shell interface, we can try to mimic the behavior of an external program and make sure our own code reacts to its output and exit codes as expected.
For the purpose of this example, consider we’re building a tiny process inspector: it has to list the processes running in the operating system.
As far as I know, there is no function for that in Go standard library, so we’ll resort to calling a widely known ps program.
First, let’s create a mock (or is it a stub?) for Shell that can be given any behavior we need:
type MockShell struct {
// an output and error to be returned when command is executed
Output []byte
Err error
// store the last executed command
// in case we need to test what command string did the code produce
LastCommand string
}
func (t *MockShell) Execute(ctx context.Context, cmd string) ([]byte, error) {
t.LastCommand = cmd
return t.Output, t.Err
}
Now, let’s implement process inspector that will make use of a previously declared Shell interface.
type ProcessInspector struct {
shell Shell
}
func (p *ProcessInspector) GetProcesses(all bool) ([]string, error) {
cmd := "ps"
if all {
cmd += " -a"
}
output, err := p.shell.Execute(context.Background(), cmd)
if err != nil {
return nil, err
}
// extract the list of processes from `ps` output
lines := strings.Split(strings.TrimSpace(string(output)), "\n")
processes := []string{}
for i, line := range lines {
if i == 0 {
// skip the header line from `ps`
continue
}
processes = append(processes, strings.TrimSpace(line))
}
return processes, nil
}
Finally, let’s add a simple unit test with a MockShell:
func TestProcessInspector_GetProcesses_all(t *testing.T) {
// Prepare a mock shell with an example output from "ps -a".
// I have copied a part of the exact output of this command from my computer.
shell := MockShell{
Output: []byte(`
PID TTY TIME CMD
1755 tty2 00:04:38 Xorg
1788 tty2 00:00:00 gnome-session-b
`),
Err: nil,
}
// instantiate process inspector with a mock shell
inspector := ProcessInspector{shell: &shell}
processes, err := inspector.GetProcesses(true)
if err != nil {
t.Fatal(err)
}
if shell.LastCommand != "ps -a" {
t.Errorf("expected the executed command to be 'ps -a', got '%s'", shell.LastCommand)
}
if len(processes) != 2 {
t.Errorf("expected 2 processes from ps -a, got %d", len(processes))
}
}
We can also add more test cases to see how GetProcesses behaves when there is an error or unexpected output.
These tests are deterministic and are fully controlled by the programmer, which is what we wanted in the first place.
Remote command execution over SSH
Life goes on and requirements change. What if the process inspector now needs to support remote execution over SSH?
Thanks to Shell interface, this feature is easy to add.
For this post, it doesn’t matter how exactly do we establish an SSH connection. I’ll use melbahja/goph package and implement a Shell interface with it.
import "github.com/melbahja/goph"
// This struct wraps a melbahja/goph SSH client and implements our Shell interface
type RemoteShell struct {
conn *goph.Client
}
// Creates an SSH client using melbahja/goph library.
// Assumes we authenticate via key file with no password.
func NewRemoteShell(host, user, keyFile string) (*RemoteShell, error) {
shell := RemoteShell{}
auth, err := goph.Key(keyFile, "")
if err != nil {
return nil, err
}
shell.conn, err = goph.NewConn(&goph.Config{
Auth: auth,
User: user,
Addr: host,
Port: 22,
Timeout: time.Second * 2,
Callback: ssh.InsecureIgnoreHostKey(),
})
if err != nil {
return nil, err
}
return &shell, nil
}
// Executes a command over SSH.
// "Goph" library provides exactly the function we need, and we just wrap it here.
func (r *RemoteShell) Execute(ctx context.Context, cmd string) ([]byte, error) {
sshCmd, err := r.conn.CommandContext(ctx, cmd)
if err != nil {
return nil, err
}
return sshCmd.CombinedOutput()
}
If you have a server available to connect to, this code can be executed right away:
package main
func main() {
remoteShell, err := NewRemoteShell("remote-address", "root", "/home/.ssh/id_rsa")
if err != nil {
log.Fatalln(err)
}
processInspector := ProcessInspector{shell: remoteShell}
processes, err := processInspector.GetProcesses(true)
if err != nil {
log.Fatalln(err)
}
fmt.Printf("%+v\n", processes)
}
As you can see, we didn’t even have to modify ProcessInspector for it to work over SSH.
Of course, the remote connection part still should be tested separately if possible, but that is just an implementation detail.
The primary part of our program doesn’t need to know what kind of Shell it uses.
Limitations
This solution is not a silver bullet. I can see why Golang developers haven’t included a Shell-like interface in a standard library.
First, it is not truly cross-platform. If your app should support multiple operating systems, this approach may not work well.
Additionally, our Shell interface is very limited as compared to Go’s exec package. It does not support pipes, redirection, output streaming, and other features on a language level.
We offloaded some of those responsibilities to sh, but then again, it is not cross-platform.
Finally, mocking shell command output is not very reliable. A developer has to put some effort into collecting all kinds of outputs manually and add test cases for them. For complex software, this is difficult or nearly impossible. Moreover, said software can change its behavior unexpectedly (for example, after an upgrade), and unit tests cannot possibly capture that.
Regardless of these limitations, I still find this approach valuable when working on server-side utilities for linux. Maybe someone else reading this can find it useful too!
The code from this post with more examples is on github: https://github.com/antonsergeyev/golang-shell-testing.