Introduction

Cmdtest is a unit testing framework for testing commands (executable programs). In other test frameworks the "unit" tested is often a class (e.g. in Java's JUnit or Ruby's Test::Unit), but in Cmdtest the unit is an executable. Apart from this difference Cmdtest borrows many ideas from the other frameworks. The program cmdtest runs the tests and reports the success or failure in different ways, e.g. by writing to standard output or producing an XML-file on Ant/JUnit format. The testcases are written in Ruby code. Assertions can be made about the side effects performed by a command:

• the exit status
• the content of standard output
• the content of standard error
• newly created/removed/changed files, or other changes to the filesystem
• encoding of a file (eg. UTF-8)

A simple example

$ cat CMDTEST_example.rb
class CMDTEST_example < Cmdtest::Testcase

  def test_hello_world
    cmd "echo hello" do
      stdout_equal "hello\n"
    end

    cmd "echo world" do
      stdout_equal "world\n"
    end
  end

  def test_touch_and_exit
    cmd "touch foo.txt ; exit 7" do
      created_files "foo.txt"
      exit_status 7
    end
  end

end

This example shows the basic structure of a testcase file. First we make a subclass of Cmdtest::Testcase. All methods of the new class with a name like test_* will be considered testcases. Inside a method we can call the cmd method. It will execute the command given as argument and then check the assertions given in the do-block. When cmdtest is run, it will find all CMDTEST_*.rb files in the current directory and run the tests in the files (see the Invoking cmdtest section for more details on CMDTEST_*.rb file selection). The output looks like:

$ cmdtest
### ======================================== CMDTEST_example.rb
### ---------------------------------------- CMDTEST_example
### ........................................ test_hello_world
### echo hello
### echo world
### ........................................ test_touch_and_exit
### touch foo.txt ; exit 7

### 1 test classes, 2 test methods, 3 commands, 0 errors, 0 fatals.

If we change "7" to "8", "foo" to "bar" and "world" to "WORLD" in the example, we get the following errors:

$ cmdtest
### ======================================== CMDTEST_example.rb
### ---------------------------------------- CMDTEST_example
### ........................................ test_hello_world
### echo hello
### echo WORLD
--- ERROR: wrong stdout
---        actual: WORLD
---        expect: world
### ........................................ test_touch_and_exit
### touch bar.txt ; exit 8
--- ERROR: created files
---        actual: ["bar.txt"]
---        expect: ["foo.txt"]
--- ERROR: expected 7 exit status, got 8

--- 1 test classes, 2 test methods, 3 commands, 2 errors, 0 fatals.

The following sections will describe in more detail what can be done with Cmdtest. See also the examples directory of the Cmdtest project, where some larger examples of Cmdtest usage can be found.

Reporting format

Normally Cmdtest writes lines on standard output to show the progress of the testing. As long as no error occurs, the lines will be prefixed by "###". Error messages will instead have a "---" prefix. This makes it easy to spot errors just by looking in the left margin. Each call to cmd will give one line on standard output. Normally the command executed will be shown (after the "###" prefix). But one can also replace the string written by calling the comment method inside the do-block of a cmd call.

When an error occurs in a test-method, the rest of the method will be skipped. But all errors occurring at the same command will be reported.

Cmdtest can also be directed to write an XML file on the same format as that used by Ant/JUnit. This makes it possible to use Cmdtest together with continuous integration servers like Jenkins.

The exit status of cmdtest will be non-zero if some errors occurred, otherwise zero. If errors should not affect exit code, the command line option --no-exit-code can be used.

Structure of a test-file

Each test-file can contain one or more subclasses to Cmdtest::Testcase. The methods that are special are:

test_*

These are the methods that will run tests. For each method, a newly created object of the class will be used.

setup

This method is called before each test_* method is called. It gives the user a chance to initialize the "environment" of all the test_* methods of the class. It can be seen as a "user level" constructor.

teardown

This method is called after each test_* method was called. It gives the user a chance to cleanup the "environment" of all the test_* methods of the class, e.g. release some resource acquired by the setup method. It can be seen as a "user level" destructor.

Structure of a test-method

Each test-method (named test_*) should contain a number of calls to the cmd method. Inside the do-block of the cmd calls, a number of assertions can be made about the outcome of the command. The simplest possible call looks like:

cmd "true" do
end

Here no explicit assertions have been given. In that case Cmdtest applies some implicit assertions. The code above is equivalent to the following more explicit one:

cmd "true" do
  exit_zero
  stdout_equal ""
  stderr_equal ""
  created_files []
  changed_files []
  removed_files []
end

The idea is that all differences in behaviour from the trivial true command should be described as an assertion in the do-block. The list of possible assertions includes: exit_zero, exit_nonzero, exit_status, created_files, changed_files, removed_files, written_files, affected_files, file_equal, file_encoding, stdout_equal and stderr_equal.

In addition to the assertions there are other helper-functions to set up the "environment" for the commands and assertions. An example is the creation of files:

...
create_file "foo.txt", "abc\ndef\n"
cmd "cat -n foo.txt" do
  stdout_equal [
    "     1\tabc",
    "     2\tdef",
  ]
end
...

The list of such helper functions includes: create_file, touch_file, import_file , import_directory and ignore_file. Beside these methods the test can of course also contain arbitrary Ruby-code.

Some test are not applicable under all circumstances. A test can for example be Linux specific. Then the function skip_test can be used, like this:

def test_linux_stuff
  if RUBY_PLATFORM !~ /linux/
    skip_test "not on linux"
  end
  ... the actual tests ...
end

If an external program tested by Cmdtest detects that the test should have been skipped, it can signal this back to Cmdtest by writing something like this on stdout or stderr:

CMDTEST_SKIP: reason for skipping test

The text "CMDTEST_SKIP" must occur at the start of a line in stdout/stderr to be recognized as a "skip order" to Cmdtest.

cmd vs shell

Sometimes one or more commands need to be run, before the interesting parts are tested. Suppose for example that we want to test that a program returning 17 from main() will exit with that status:

create_file "exit17.c", [
    "extern int status;",
    "int main() { return status; }\n",
]
create_file "status.c", [
    "int status = 17;",
]
cmd "gcc -c exit17.c status.c" do
    created_files "exit17.o", "status.o"
end
cmd "gcc -o exit17 exit17.o status.o" do
    created_files "exit17"
end
cmd "./exit17" do
    exit_status 17
end

The fact that the files exit17.o, status.o and exit17 are created in the two first steps are uninteresting here. To make it easier to create such an initial setup, one can use the command shell instead or cmd:

create_file "exit17.c", [
    "extern int status;",
    "int main() { return status; }\n",
]
create_file "status.c", [
    "int status = 17;",
]
shell "gcc -c exit17.c status.c"
shell "gcc -o exit17 exit17.o status.o"
cmd "./exit17" do
    exit_status 17
end

The shell command does not care about the "side effects" of the command, as long as the command terminates with zero exit status. But if such a command fails, it will output its stdout/stderr. If for example the variable "status" above was misspelled in status.c one could get:

--- ERROR: expected zero exit status, got 1
--- INFO: the stdout
---        actual: [[empty]]
--- INFO: the stderr
---        actual: /usr/bin/ld: exit17.o: in function `main':
---                exit17.c:(.text+0xa): undefined reference to `status'
---                collect2: error: ld returned 1 exit status

Work directory

All tests are performed in a "clean" temporary directory, here called the "work directory". When the setup, test_* and teardown methods are called the current directory will be the "work directory" (unless chdir is called by the methods themselves).

Several of the assertions and helper functions take filename arguments that are evaluated relative to the "work directory" (or sometimes the current directory if they differ).

Cmdtest implements parallel execution of test methods by running several "slave processes", started by a tool like GNU Parallel.

Methods such as File.open and Dir.chdir that depend on the "current directory" can be used in the test methods, since each slave is a process of its own (an earlier version of Cmdtest used Ruby threads and advised against using such methods).

Specifying files / directories

Several methods take files or directories as argument (e.g. created_files, modified_files and ignore_file). Instead of having two sets of methods, one for files and one for directories, an argument with a trailing "/" denotes a directory:

created_files "build/"       # the directory "build"
created_files "build"        # the file "build"

ignore_file "build/"        # the directory "build" (and everything below)
ignore_file "build"         # the file "build"

As can be seen in the example above, the ignore_file method is special, because an ignored directory means that all files below the directory are ignored too. Other peculiarities of ignore_file are that the argument can be a Regexp or shell wildcards:

ignore_file /\.o$/          # all files *.o (in subdirs too)
ignore_file "**/*.o"        # all files *.o (in subdirs too)

This is quite natural, since the "job" of ignore_file is to single out a subset of all files.

PATH handling

Cmdtest is used to test commands, so an important question is how the commands are found and executed. Normally commands are found via the PATH environment variable, and Cmdtest is no exception. The commands executed in the cmd calls are evaluated in a shell script (on UN*X) or in a BAT file (on Windows). The PATH in effect when cmdtest is invoked is kept intact, with one addition: the current directory at the time of invocation is prepended to the PATH. If further changes to the PATH are needed the methods prepend_path, prepend_local_path or set_path can be used. Such path modifications does not survive between test methods. Each new test method starts with the original value of PATH.

Matching standard output content

An assertion like stdout_equal compares the actual standard output of a command with the expected outcome. The expected value can be specified in different ways, and is best explained by example:

cmd "echo hello ; echo world" do
  stdout_equal "hello\nworld\n"            # 1
  stdout_equal [                           # 2
    "hello",
    "world"
  ]
  stdout_equal /orld/                      # 3
  stdout_equal [                           # 4
    "hello",
    /world|earth/
  ]
end

In the example we see how the content can be specified:

1. as a string, with a newline (\n) character for each new line
2. as an array of lines
3. as a regexp that should match the file content given as a string
4. as an array of lines where some lines should match a regexp rather than be compared for string equality

$ cat CMDTEST_newline.rb
class CMDTEST_newline < Cmdtest::Testcase
  def test_newline
    cmd 'echo -n "hello\r\nworld\r\n"' do
      exit_zero
      stdout_equal [ "hello", "world" ]
    end
  end
end
$
$ cmdtest -q CMDTEST_newline.rb
### echo -n "hello\r\nworld\r\n"
--- ERROR: Windows line ending: STDOUT

cmd 'echo -n "hello\r\nworld\r\n"' do
  exit_zero
  output_newline "\r\n" do
    stdout_equal [ "hello", "world" ]
  end
end

output_encoding "utf-8" do
  ...
end

Invoking cmdtest

cmdtest can be called without any arguments at all. It will then look for CMDTEST_*.rb files in the following places:

1. first t/CMDTEST_*.rb
2. second test/CMDTEST_*.rb
3. otherwise CMDTEST_*.rb

If some command line arguments have been given, cmdtest will use them instead of searching by itself. Some examples:

$ cmdtest CMDTEST_foo.rb                   # just one file
$ cmdtest CMDTEST_foo.rb CMDTEST_bar.rb    # two files
$ cmdtest t                                # all CMDTEST_*.rb files in "t" dir
$ cmdtest . t                              # all CMDTEST_*.rb files in both dirs

In addition to test files, the command line can also contain options and testcase selectors. The general format is:

$ cmdtest [options] [files] [selectors]

The options are describe in a separate section below. The selectors are regular expressions that are used to match the names of the test methods. It is best illustrated by an example:

$ cmdtest examples "/stdin|stdout/"

This command will find all files matching examples/CMDTEST_*.rb, and run all test methods whose names either contain the string "stdin" or "stdout". As can be seen in the example, the regular expression may need protection from expansion by the shell (that is the reason for the quotes in the example). But the example can also be written:

$ cmdtest examples /stdin/ /stdout/

For more examples of command line usage, see the section Commandline Examples below.

$ cmdtest -h
usage: cmdtest [-h] [--shortversion] [--version] [-q] [-v] [--diff] [--no-diff]
               [--fast] [-j N] [--test TEST] [--xml FILE] [--no-exit-code]
               [--stop-on-error] [-i] [--slave SLAVE]
               [arg [arg ...]]

positional arguments:
  arg           testfile or pattern

optional arguments:
  -h, --help            show this help message and exit
  --shortversion        show just version number
  --version             show version
  -q, --quiet           be more quiet
  -v, --verbose         be more verbose
  --diff                diff output (default)
  --no-diff             old non-diff output
  --fast                run fast without waiting for unique mtime:s
  -j N, --parallel N    build in parallel
  --test TEST           only run named test
  --xml FILE            write summary on JUnit format
  --no-exit-code        exit with 0 status even after errors
  --stop-on-error       exit after first error
  -i, --incremental     incremental mode
  --slave SLAVE         run in slave mode

$ cmdtest

$ cmdtest -i

$ cmdtest /stdout/

$ cmdtest examples

$ cmdtest --xml=reports/test-foo.xml

Reference Part

cmd "some_prog ..." do
  assertion1 ...
  ...
  assertionN ...
end

# all assertions implicit
cmd "true" do
end

# exit status assertion explicit, but other assertions implicit
cmd "true" do
  exit_zero
end