Cmdtest User Guide ================== .. contents:: Introduction ------------ Cmdtest is a `unit testing`_ framework for testing commands (executable programs). In other test frameworks the "unit" tested is often a class (e.g. in Java's JUnit_ or Ruby's `Test::Unit`_), but in Cmdtest the unit is an executable. Apart from this difference Cmdtest borrows many ideas from the other frameworks. The program ``cmdtest`` runs the tests and reports the success or failure in different ways, e.g. by writing to standard output or producing an XML-file on Ant/JUnit format. The testcases are written in Ruby code. Assertions can be made about the side effects performed by a command: - the exit status - the content of standard output - the content of standard error - newly created/removed/changed files, or other changes to the filesystem - encoding of a file (eg. UTF-8) A simple example ---------------- :: $ cat CMDTEST_example.rb class CMDTEST_example < Cmdtest::Testcase def test_hello_world cmd "echo hello" do stdout_equal "hello\n" end cmd "echo world" do stdout_equal "world\n" end end def test_touch_and_exit cmd "touch foo.txt ; exit 7" do created_files "foo.txt" exit_status 7 end end end This example shows the basic structure of a testcase file. First we make a subclass of ``Cmdtest::Testcase``. All methods of the new class with a name like ``test_*`` will be considered testcases. Inside a method we can call the ``cmd`` method. It will execute the command given as argument and then check the assertions given in the do-block. When ``cmdtest`` is run, it will find all ``CMDTEST_*.rb`` files in the current directory and run the tests in the files (see the `Invoking cmdtest`_ section for more details on ``CMDTEST_*.rb`` file selection). The output looks like:: $ cmdtest ### ======================================== CMDTEST_example.rb ### ---------------------------------------- CMDTEST_example ### ........................................ test_hello_world ### echo hello ### echo world ### ........................................ test_touch_and_exit ### touch foo.txt ; exit 7 ### 1 test classes, 2 test methods, 3 commands, 0 errors, 0 fatals. If we change "7" to "8", "foo" to "bar" and "world" to "WORLD" in the example, we get the following errors:: $ cmdtest ### ======================================== CMDTEST_example.rb ### ---------------------------------------- CMDTEST_example ### ........................................ test_hello_world ### echo hello ### echo WORLD --- ERROR: wrong stdout --- actual: WORLD --- expect: world ### ........................................ test_touch_and_exit ### touch bar.txt ; exit 8 --- ERROR: created files --- actual: ["bar.txt"] --- expect: ["foo.txt"] --- ERROR: expected 7 exit status, got 8 --- 1 test classes, 2 test methods, 3 commands, 2 errors, 0 fatals. The following sections will describe in more detail what can be done with Cmdtest. See also the `examples directory <../examples>`_ of the Cmdtest project, where some larger examples of Cmdtest usage can be found. Reporting format ---------------- Normally Cmdtest writes lines on standard output to show the progress of the testing. As long as no error occurs, the lines will be prefixed by "###". Error messages will instead have a "---" prefix. This makes it easy to spot errors just by looking in the left margin. Each call to ``cmd`` will give one line on standard output. Normally the command executed will be shown (after the "###" prefix). But one can also replace the string written by calling the ``comment`` method inside the do-block of a ``cmd`` call. When an error occurs in a test-method, the rest of the method will be skipped. But all errors occurring at the same command will be reported. Cmdtest can also be directed to write an XML file on the same format as that used by Ant/JUnit. This makes it possible to use Cmdtest together with `continuous integration`_ servers like Jenkins_. The exit status of ``cmdtest`` will be non-zero if some errors occurred, otherwise zero. If errors should not affect exit code, the command line option ``--no-exit-code`` can be used. Structure of a test-file ------------------------ Each test-file can contain one or more subclasses to ``Cmdtest::Testcase``. The methods that are special are: ``test_*`` These are the methods that will run tests. For each method, a newly created object of the class will be used. ``setup`` This method is called before each ``test_*`` method is called. It gives the user a chance to initialize the "environment" of all the ``test_*`` methods of the class. It can be seen as a "user level" constructor. ``teardown`` This method is called after each ``test_*`` method was called. It gives the user a chance to cleanup the "environment" of all the ``test_*`` methods of the class, e.g. release some resource acquired by the ``setup`` method. It can be seen as a "user level" destructor. Structure of a test-method -------------------------- Each test-method (named ``test_*``) should contain a number of calls to the ``cmd`` method. Inside the do-block of the ``cmd`` calls, a number of assertions can be made about the outcome of the command. The simplest possible call looks like:: cmd "true" do end Here no explicit assertions have been given. In that case Cmdtest applies some implicit assertions. The code above is equivalent to the following more explicit one:: cmd "true" do exit_zero stdout_equal "" stderr_equal "" created_files [] changed_files [] removed_files [] end The idea is that all differences in behaviour from the trivial ``true`` command should be described as an assertion in the do-block. The list of possible assertions includes: ``exit_zero``, ``exit_nonzero``, ``exit_status``, ``created_files``, ``changed_files``, ``removed_files``, ``written_files``, ``affected_files``, ``file_equal``, ``file_encoding``, ``stdout_equal`` and ``stderr_equal``. In addition to the assertions there are other helper-functions to set up the "environment" for the commands and assertions. An example is the creation of files:: ... create_file "foo.txt", "abc\ndef\n" cmd "cat -n foo.txt" do stdout_equal [ " 1\tabc", " 2\tdef", ] end ... The list of such helper functions includes: ``create_file``, ``touch_file``, ``import_file`` , ``import_directory`` and ``ignore_file``. Beside these methods the test can of course also contain arbitrary Ruby-code. Some test are not applicable under all circumstances. A test can for example be Linux specific. Then the function ``skip_test`` can be used, like this:: def test_linux_stuff if RUBY_PLATFORM !~ /linux/ skip_test "not on linux" end ... the actual tests ... end If an external program tested by Cmdtest detects that the test should have been skipped, it can signal this back to Cmdtest by writing something like this on stdout or stderr:: CMDTEST_SKIP: reason for skipping test The text ``"CMDTEST_SKIP"`` must occur at the start of a line in stdout/stderr to be recognized as a "skip order" to Cmdtest. cmd vs shell ------------ Sometimes one or more commands need to be run, before the interesting parts are tested. Suppose for example that we want to test that a program returning 17 from ``main()`` will exit with that status:: create_file "exit17.c", [ "extern int status;", "int main() { return status; }\n", ] create_file "status.c", [ "int status = 17;", ] cmd "gcc -c exit17.c status.c" do created_files "exit17.o", "status.o" end cmd "gcc -o exit17 exit17.o status.o" do created_files "exit17" end cmd "./exit17" do exit_status 17 end The fact that the files ``exit17.o``, ``status.o`` and ``exit17`` are created in the two first steps are uninteresting here. To make it easier to create such an initial setup, one can use the command ``shell`` instead or ``cmd``:: create_file "exit17.c", [ "extern int status;", "int main() { return status; }\n", ] create_file "status.c", [ "int status = 17;", ] shell "gcc -c exit17.c status.c" shell "gcc -o exit17 exit17.o status.o" cmd "./exit17" do exit_status 17 end The ``shell`` command does not care about the "side effects" of the command, as long as the command terminates with zero exit status. But if such a command fails, it will output its stdout/stderr. If for example the variable "status" above was misspelled in ``status.c`` one could get:: --- ERROR: expected zero exit status, got 1 --- INFO: the stdout --- actual: [[empty]] --- INFO: the stderr --- actual: /usr/bin/ld: exit17.o: in function `main': --- exit17.c:(.text+0xa): undefined reference to `status' --- collect2: error: ld returned 1 exit status Work directory -------------- All tests are performed in a "clean" temporary directory, here called the "work directory". When the ``setup``, ``test_*`` and ``teardown`` methods are called the current directory will be the "work directory" (unless ``chdir`` is called by the methods themselves). Several of the assertions and helper functions take filename arguments that are evaluated relative to the "work directory" (or sometimes the current directory if they differ). Cmdtest implements parallel execution of test methods by running several "slave processes", started by a tool like `GNU Parallel`_. Methods such as ``File.open`` and ``Dir.chdir`` that depend on the "current directory" can be used in the test methods, since each slave is a process of its own (an earlier version of Cmdtest used Ruby threads and advised against using such methods). Specifying files / directories ------------------------------ Several methods take files or directories as argument (e.g. ``created_files``, ``modified_files`` and ``ignore_file``). Instead of having two sets of methods, one for files and one for directories, an argument with a trailing "/" denotes a directory:: created_files "build/" # the directory "build" created_files "build" # the file "build" ignore_file "build/" # the directory "build" (and everything below) ignore_file "build" # the file "build" As can be seen in the example above, the ``ignore_file`` method is special, because an ignored directory means that all files below the directory are ignored too. Other peculiarities of ``ignore_file`` are that the argument can be a Regexp or shell wildcards:: ignore_file /\.o$/ # all files *.o (in subdirs too) ignore_file "**/*.o" # all files *.o (in subdirs too) This is quite natural, since the "job" of ``ignore_file`` is to single out a subset of all files. PATH handling ------------- Cmdtest is used to test commands, so an important question is how the commands are found and executed. Normally commands are found via the ``PATH`` environment variable, and Cmdtest is no exception. The commands executed in the ``cmd`` calls are evaluated in a shell script (on UN*X) or in a BAT file (on Windows). The ``PATH`` in effect when ``cmdtest`` is invoked is kept intact, with one addition: the current directory at the time of invocation is prepended to the ``PATH``. If further changes to the ``PATH`` are needed the methods ``prepend_path``, ``prepend_local_path`` or ``set_path`` can be used. Such path modifications does not survive between test methods. Each new test method starts with the original value of ``PATH``. Matching standard output content -------------------------------- An assertion like ``stdout_equal`` compares the actual standard output of a command with the expected outcome. The expected value can be specified in different ways, and is best explained by example:: cmd "echo hello ; echo world" do stdout_equal "hello\nworld\n" # 1 stdout_equal [ # 2 "hello", "world" ] stdout_equal /orld/ # 3 stdout_equal [ # 4 "hello", /world|earth/ ] end In the example we see how the content can be specified: 1) as a string, with a newline (``\n``) character for each new line 2) as an array of lines 3) as a regexp that should match the file content given as a string 4) as an array of lines where some lines should match a regexp rather than be compared for string equality Unusual newlines ++++++++++++++++ Some programs produce text output with unusual newlines for the current platform, for example ``"\n"`` on Windows or ``"\r\n"`` on Linux. The following test will give an error on Linux:: $ cat CMDTEST_newline.rb class CMDTEST_newline < Cmdtest::Testcase def test_newline cmd 'echo -n "hello\r\nworld\r\n"' do exit_zero stdout_equal [ "hello", "world" ] end end end $ $ cmdtest -q CMDTEST_newline.rb ### echo -n "hello\r\nworld\r\n" --- ERROR: Windows line ending: STDOUT To cope with this situation, it is possible to use the command ``output_newline`` like this:: cmd 'echo -n "hello\r\nworld\r\n"' do exit_zero output_newline "\r\n" do stdout_equal [ "hello", "world" ] end end The argument to ``output_newline`` can be one of ``"\n"``, ``"\r\n"``, or ``:consistent``. The last one means either newline is ok, as long as all lines use the same newline. The call can use a do-block as in the example above. A plain method call without do-block can also be used, and then the new setting is in effect until it is set again, or the test method ends. Unusual text encoding +++++++++++++++++++++ In the same manner as for newline, the encoding can be something other than the default (in this case ``"ascii"``). For example:: output_encoding "utf-8" do ... end The argument should be an encoding name known to Ruby. Invoking ``cmdtest`` -------------------- ``cmdtest`` can be called without any arguments at all. It will then look for ``CMDTEST_*.rb`` files in the following places: 1) first ``t/CMDTEST_*.rb`` 2) second ``test/CMDTEST_*.rb`` 3) otherwise ``CMDTEST_*.rb`` If some command line arguments have been given, ``cmdtest`` will use them instead of searching by itself. Some examples:: $ cmdtest CMDTEST_foo.rb # just one file $ cmdtest CMDTEST_foo.rb CMDTEST_bar.rb # two files $ cmdtest t # all CMDTEST_*.rb files in "t" dir $ cmdtest . t # all CMDTEST_*.rb files in both dirs In addition to test files, the command line can also contain options and testcase selectors. The general format is:: $ cmdtest [options] [files] [selectors] The options are describe in a separate section below. The selectors are regular expressions that are used to match the names of the test methods. It is best illustrated by an example:: $ cmdtest examples "/stdin|stdout/" This command will find all files matching ``examples/CMDTEST_*.rb``, and run all test methods whose names either contain the string "stdin" or "stdout". As can be seen in the example, the regular expression may need protection from expansion by the shell (that is the reason for the quotes in the example). But the example can also be written:: $ cmdtest examples /stdin/ /stdout/ For more examples of command line usage, see the section `Commandline Examples`_ below. Options +++++++ The available options can be seen by using the ``-h`` option:: $ cmdtest -h usage: cmdtest [-h] [--shortversion] [--version] [-q] [-v] [--diff] [--no-diff] [--fast] [-j N] [--test TEST] [--xml FILE] [--no-exit-code] [--stop-on-error] [-i] [--slave SLAVE] [arg [arg ...]] positional arguments: arg testfile or pattern optional arguments: -h, --help show this help message and exit --shortversion show just version number --version show version -q, --quiet be more quiet -v, --verbose be more verbose --diff diff output (default) --no-diff old non-diff output --fast run fast without waiting for unique mtime:s -j N, --parallel N build in parallel --test TEST only run named test --xml FILE write summary on JUnit format --no-exit-code exit with 0 status even after errors --stop-on-error exit after first error -i, --incremental incremental mode --slave SLAVE run in slave mode Commandline Examples ++++++++++++++++++++ This section is a collection of examples of how Cmdtest can be used. :: $ cmdtest This is the most basic usage. All testcase files found (by the algorithm described earlier) will be executed. :: $ cmdtest -i Only run the test methods that have failed earlier, or have changed. This is not a full-blown "make system", but may still be useful when developing the tests. :: $ cmdtest /stdout/ Run all test methods matching the regular expression given. :: $ cmdtest examples Run all tests found in test files in the "examples" directory (i.e. ``examples/CMDTEST_*.rb``). :: $ cmdtest --xml=reports/test-foo.xml Write an XML-summary to the specified file. The file uses the same format as JUnit_, so it can be understood be continuous integration servers such as Jenkins_. Reference Part -------------- cmd +++ The ``cmd`` method is the central method of the whole Cmdtest framework. It should always be called with a block like this:: cmd "some_prog ..." do assertion1 ... ... assertionN ... end A block is used to make it easy to know when the last assertion has been found. The do-block should only contain assertions. Cmdtest applies some implicit assertions if the do-block is empty or misses some kind of assertion:: # all assertions implicit cmd "true" do end # exit status assertion explicit, but other assertions implicit cmd "true" do exit_zero end See also the example in the `Structure of a test-method`_ section above. shell +++++ The ``shell`` method is similar to ``cmd``, but with some important differences: - ``shell`` only cares about the exit status - stdout/stderr and any created files are ignored - it is called without any "Ruby do block" - if the exit status is non-zero, stdout/stderr is reported for information purposes For more details see `cmd vs shell`_. Assertions - exit status ++++++++++++++++++++++++ These methods should only be used inside a ``cmd`` block. ``exit_nonzero`` The command should have exited with a non-zero exit status (i.e. it should have failed). ``exit_status(status)`` The command should have exited with the specified exit status. ``exit_zero`` The command should have exited with a zero exit status (i.e. it should have succeeded). This is the default if none of the other exit-related methods have been called. Assertions - files ++++++++++++++++++ These methods should only be used inside a ``cmd`` block. ``affected_files(file1,...,fileN)`` The specified files should have been created, removed or modified by the command. This assertion can be used when it doesn't matter which of ``created_files``, ``removed_files`` or ``changed_files`` that apply (cf. ``written_files``). ``created_files(file1,...,fileN)`` The specified files should have been created by the command. ``modified_files(file1,...,fileN)`` The specified files should have been modified by the command. A file is considered modified if it existed before the command, and something about the file has changed after the command (inode number, modification date or content). ``removed_files(file1,...,fileN)`` The specified files should have been removed by the command. ``written_files(file1,...,fileN)`` The specified files should have been created or modified by the command. This assertion can be used when it doesn't matter which of ``created_files`` or ``changed_files`` that apply. A typical scenario is in a test method where repeated operations are done on the same file. By using ``written_files`` we don't have to treat the first case special (when the file is created). Assertions - stdout/stderr/file content +++++++++++++++++++++++++++++++++++++++ These methods should only be used inside a ``cmd`` block. ``file_equal(file, content)`` Assert that the specified file matches the given content. See "stdout_equal" for how "content" can be specified. ``file_not_equal(file, content)`` Like ``file_equal`` but with inverted test. ``file_encoding(file, enc, bom: nil)`` Assert that the file uses encoding ``enc``. This is verified by reading the file using that encoding. The optional ``bom`` argument can be used to assert the existence/non-existence of a Unicode BOM. ``stderr_equal(content)`` Assert that the standard error of the command matches the given content. See "stdout_equal" for how "content" can be specified. ``stderr_not_equal(content)`` Like ``stderr_equal`` but with inverted test. ``stdout_equal(content)`` Assert that the standard output of the command matches the given content. The content can be given in several different forms: 1) as a string that should be equal to the entire file, 2) as an array of lines that should be equal to the entire file, 3) as a regexp that should match the entire file (given as one string). For more details and examples see the section `Matching standard output content`_. ``stdout_not_equal(content)`` Like ``stdout_equal`` but with inverted test. Assertions - misc +++++++++++++++++ These methods should only be used inside a ``cmd`` block. ``assert(flag, msg=nil)`` Assert that ``flag`` is true. This assertion is a last resort, when no other assertion fits. Should normally not be used. ``time(interval)`` Assert that executing the command took a number of seconds inside the interval given as argument. Helper functions ++++++++++++++++ These methods should only be used outside a ``cmd`` block in a test method, or in the ``setup`` method. ``comment(msg)`` Normally the command executed by a ``cmd`` call, will be shown on STDOUT (after the usual "###" prefix). But one can also replace the string written by calling the ``comment`` method inside the do-block of a ``cmd`` call. Then the ``comment`` call should occur before method calls inspecting the result of the executed command. ``create_file(filename, content)`` Create a file inside the "work directory". If the filename contains a directory part, intermediate directories are created if needed. The content can be specified either as an array of lines or as a string with the content of the whole file. The filename is evaluated relative to the current directory at the time of the call. ``dont_ignore_files(file1, ..., fileN)`` Don't ignore the specified files when looking for differences in the filesystem. This overrides a previous call to ``ignore_files``. If a previous call to ``ignore_files`` ignored a whole directory tree, the call to ``dont_ignore_files`` can reverse the effect for specific files inside that directory tree. ``ignore_file(file)`` Ignore the specified file when looking for differences in the filesystem. A subdirectory and anything in it can be ignored by giving a trailing "/" to the name. Shell wildcard can be used (``"*"`` and ``"**"``). A Regexp argument is matched against the path of files. ``ignore_file`` is typically used at the beginning of a test method, before any calls to ``cmd``. But ``ignore_file`` can also be used at the beginning of a ``cmd`` do block before method calls inspecting the result of the executed command. In this case the effect of the call is local to just that do block. ``ignore_files(file1, ..., fileN)`` Ignore the specified files when looking for differences in the filesystem. See ``ignore_file`` for details. ``ignore_stdout_stderr()`` Ignore the stdout and stderr from the command. If the exit status of the command is unexpected, the actual stdout and stderr is still shown, prefixed with "INFO", since the output can contain some interesting hints why the exit status was unexpected. ``import_file(src, tgt)`` Copy a file from outside of the "work directory" to inside. The ``src`` path is evaluated relative to the current directory when ``cmdtest`` was called. The ``tgt`` is evaluated relative to the current directory inside the "work directory" at the time of the call. ``import_directory(src, tgt)`` Copy a directory tree from outside of the "work directory" to inside. The ``src`` path is evaluated relative to the current directory when ``cmdtest`` was called. The ``tgt`` is evaluated relative to the current directory inside the "work directory" at the time of the call. It is an error if ``tgt`` already exists. ``prepend_local_path(dir)`` Prepend the given directory to the ``PATH`` so commands executed via ``cmd`` are looked up using the modified ``PATH``. The argument ``dir`` is evaluated relative to the current directory in effect at the time of the call (i.e. typically the "work directory" during the test). ``prepend_path(dir)`` Prepend the given directory to the ``PATH`` so commands executed via ``cmd`` are looked up using the modified ``PATH``. A typical use is to add the directory where the executable tested is located. The argument ``dir`` is evaluated relative to the current directory in effect when ``cmdtest`` was invoked. ``setenv(name, value)`` Set an environment variable that should be in effect when commands are executed by later calls to ``cmd``. ``unsetenv(name)`` Unset an environment variable that should not be in effect when commands are executed by later calls to ``cmd``. ``set_path(dir1, ..., dirN)`` Set ``PATH`` to the given directories, so commands executed via ``cmd`` are looked up using the modified ``PATH``. This method sets the whole ``PATH`` rather than modifying it (in contrast to ``prepend_path`` and ``prepend_local_path``). ``skip_test(reason)`` If a test method should not be run for some reason (eg. wrong platform), this can be signaled to ``cmdtest`` by calling ``skip_test`` at the beginning of the test method. The argument should mention why the test is skipped. Such tests will be reported as "skipped", and also show up in the JUnit format XML files (the intention is that this should work like the "assume-functions" in recent versions of JUnit). ``touch_file(filename)`` "touch" a file inside the "work directory". The filename is evaluated relative to the current directory at the time of the call. ``stdout_check()`` Will "callback" to the test script, giving the user a chance to inspect STDOUT, and maybe call "assert". Example:: cmd "seq 0 5 100" do stdout_check do |lines| assert lines.include?(55), "no line '55'" end end ``stdout_check()`` Will "callback" to the test script, giving the user a chance to inspect STDERR, and maybe call "assert" (see ``stdout_check()`` above). Deprecated helper functions +++++++++++++++++++++++++++ ``dir_mkdir(file)`` Deprecated, use ``Dir.mkdir`` instead. ``file_chmod(arg, file)`` Deprecated, use ``File.chmod`` instead. ``file_open(file, *args, &block)`` Deprecated, use ``File.open`` instead. ``file_read(file)`` Deprecated, use ``File.read`` instead. ``file_symlink(file1, file2)`` Deprecated, use ``File.symlink`` instead. ``file_utime(arg1, arg2, file)`` Deprecated, use ``File.utime`` instead. ``remove_file(file)`` Deprecated, use ``FileUtils.rm_f`` instead. ``remove_file_tree(file)`` Deprecated, use ``FileUtils.rm_rf`` instead. .. _`unit testing`: http://en.wikipedia.org/wiki/Unit_testing .. _`junit`: http://en.wikipedia.org/wiki/JUnit .. _`Test::Unit`: https://github.com/test-unit/test-unit .. _`continuous integration`: http://en.wikipedia.org/wiki/Continuous_integration .. _Jenkins: http://jenkins-ci.org .. _`GNU Parallel`: http://www.gnu.org/software/parallel/