README: About this repository

Table of Contents

• Installation of commandunit
• Usage
  • Command Line
  • Test Script Format
  • Default Directory Layout
• Design of commandunit
• Installation Troubleshooting
• References

This repository provides an installer of commandunit tool. Also, this contains an example project to illustrate how to use commanduunit (github) and how to organize a shell script based project using it.

To install it, just run:

git clone https://github.com/dakusui/commandunit-installer.git --branch master --single-branch --depth=1
cd commandunit-installer
./install.sh

To just run tests under src/main/scripts, type the following command from your terminal:

commandunit --clean (1)

1. commandunit generates intermediate files and this is sometimes time-consuming if you have a lot of tests. You can reuse already generated files by default. --clean cleans up them.

Note that you need a docker CLI installed, and docker daemon running. Also, we tested this repository with bash 5.1 (Ubuntu 22.04).

On a "successful" execution, it will show:

$ ./build.sh
1..8
# Processing 'yaml++' files.
ok # skip 1 process file: 'core/test-abort.yaml++'
ok # skip 2 process file: 'core/test-assert-false.yaml++'
ok 3 process file: core/test-cat-failing.yaml++)
ok # skip 4 process file: 'core/test-cat.yaml++'
ok # skip 5 process file: 'core/test-is_void_value-false.yaml++'
ok # skip 6 process file: 'core/test-is_void_value-true.yaml++'
ok # skip 7 process file: 'core/test-set.yaml++'
ok # skip 8 process file: 'core/test-void_value.yaml++'
1..0
# Processing 'yaml' files.
1..8
# Processing 'json++' files.
ok # skip 1 process file: 'core/test-abort.json++'
ok # skip 2 process file: 'core/test-assert-false.json++'
ok 3 process file: core/test-cat-failing.json++)
ok # skip 4 process file: 'core/test-cat.json++'
ok # skip 5 process file: 'core/test-is_void_value-false.json++'
ok # skip 6 process file: 'core/test-is_void_value-true.json++'
ok # skip 7 process file: 'core/test-set.json++'
ok # skip 8 process file: 'core/test-void_value.json++'
1..8
ok 1 core/test-abort.json
ok 2 core/test-assert-false.json
not ok 3 core/test-cat-failing.json
ok 4 core/test-cat.json
ok 5 core/test-is_void_value-false.json
ok 6 core/test-is_void_value-true.json
ok 7 core/test-set.json
ok 8 core/test-void_value.json
FAILED: CHECK /home/hiroshi/Documents/github/commandunit-example/commandunit-out/report/testreport.adoc
ERROR:(exit code:1, pipe status: 1):
  at /app/dependencies/bud/lib/core.rc:54 (abort)
  at /app/bin/commandunit:189 (main)

One and only one test in the example test suite should fail and the command should exit with 1. Don’t worry, the one (core/test-cat-failing.json) is designed to fail intentionally in order to illustrate how a test report looks like.

Installation of commandunit

To try the from your command line, do:

./install.sh

This will download a "wrapper" script and place it under your $HOME/bin directory. You need to have the directory in your PATH environment variable.

Usage

Command Line

commandunit has "native" and normal (docker) execution modes. The "native" mode uses command line tools installed on your system to execute commandunit 's functionality, while normal mode uses a docker image which contains all the necessary dependencies. The default is the "normal" mode. To use "native" mode, give --native option to commandunit.

"native" mode

commandunit --native

commandunit generates intermediate files to accelerate its execution. Those intermediate files will be reused if source test files are not updated by default. However, this mechanism will get confused in case the files are generated for "native" mode but reused by normal mode or vice versa. To remove those internal files, use --clean option.

--clean option

commandunit --clean

To show the help, use --help option.

Help

commandunit --help

Also, check Command Line Interface Handbook for more detail.

Test Script Format

Following is an example of a test script defined for commandunit. You can find a file in src/test/scripts/core directory.

Example: scripts/core/test-cat-yaml.yaml++

---
"$extends":
  - core/base-yaml.yaml # (1)
when: # (2)
  environmentVariables:
    SCRIPTS_DIR: "${COMMANDUNIT_PROJECT_DIR}/src/main/scripts"
  source:
    - ${COMMANDUNIT_BUILTIN_ROOT}/bud/lib/core.rc
    - ${SCRIPTS_DIR}/target_lib.rc
  cmd: cat
  args:
    - ${SCRIPTS_DIR}/hello.txt
then:
  exitCode:
    - EQUAL
    - 0
  stdout:
    present:
      - REGEX:Hello world
  stderr:
    absent:
      - REGEX:.+

1. Check $extends attribute

2. Check when clause

$extends attribute

This attribute specifies a list of yaml++, json++, or json files that this file inherits. In this example, it is specified that a file core/base-yaml.yaml. The file is stored as src/test/.commandunit/jsonpp/core/base-yaml.yaml. The file then extends base/normal.json, which is a built-in file, in the ends. When you use $extends attribute, the file in the array will be searched from src/test/.commandunit and built-in directory. The built-in templates available are found in the commandunit 's repository (built-in templates).

when clause

This attribute defines what will be exercised in a test.

description

An array. You can describe the operation to be verified in this test. This attribute is currently a placce-holder and not used by the commandunit. It may be a part of testreport.adoc in future, though.

  description:
    - Call 'abort' function define in core.rc.
    - This test is to illustrate the usage of the function.

environmentVariables

An object. You can store key-value pair whose key is a name of an environment variable. The following example defines an environment variable SCRITPS_DIR and it will be available from the command defined by cmd and args.

  environmentVariables:
    SCRIPTS_DIR: "${COMMANDUNIT_HOSTFSROOT_MOUNTPOINT}/${COMMANDUNIT_PWD}/src/main/scripts"

source

An array you can list files to be sourced before the command (cmd) is executed.

  source:
    - ${COMMANDUNIT_BUILTIN_ROOT}/bud/lib/core.rc
    - ${COMMANDUNIT_PROJECT_DIR}/src/main/scripts/target_lib.rc

In this example, a built-in library core.rc and a library under test target_lib.rc are sourced. Functions defined in those files can be executed from the cmd attribute.

cmd(a string) and args(an array of strings)

A command (or more generally an "executable", such as a function) to be executed and arguments to be passed to the command. The following example executes a function, which is defined in bud/core.rc. this example will execute a command line: assert_that HELLO [[ hello == Hello ]] .cmd and args example

  cmd: assert_that
  args:
    - HELLO
    - "[["
    - hello
    - ==
    - Hello
    - "]]"

then clause

This attribute defines how the exercised operation is verified.

description

You can describe how the operation defined in when should be verified.

exitCode

An array. You can describe the condition that should be satisfied by an exit code of cmd in when clause. The first element is a name of a predicate factory and the rest will be arguments passed to the factory. The exit code will then be checked with the predicated created by the factory. Currently available predicate factory names are following:

• EQUAL

• NOT_EQUAL

  # noinspection YAMLIncompatibleTypes
exitCode:
    - NOT_EQUAL
    - 0

stdout and stderr

An object. You can describe the expectation for stdout and stderr of cmd in when clause. Under this object you can place present and absent attributes.

present and absent

arrays. These attributes define patterns that should present or absent in the target stream (stdout or stderr). If you give a string starting with REGEX:, the rest will be treated as a regular expression. If a matching line for a pattern in present attribute is find in the target stream, the condition defined by the string will be considered satisfied. If and only if all the conditions defined for present are satisfied, the target stream will be considered "pass" for present. For absent, if and only if none of the conditions defined for it are satisfied, the tatarget stream will be considered "pass". Following is an example, where nothing should be output to stdout.

  stdout:
    absent:
      - REGEX:.+

Built-in Environment Variables

Following is a list of built-in environment variables of commandunit.

COMMANDUNIT_PROJECT_DIR

The top level directory of you project. The actual value can be different depending on whether you are using commandunit in "native" mode or not.

COMMANDUNIT_BUILTIN_ROOT

The directory under which built-ins are stored. It points a directory under {commandunit-home}/src/main/scripts/lib.

For more details, check Built-ins.

Built-in Library

The commandunit has a set of libraries to make it easy to write tests. The details of them can be found here: Built-ins

In this section, useful functions for testing will be walked through.

bud/core.rc

message

prints a message to stderr.

assert_that

tests a given condition (arguments except the first) and aborts if the condition doesn’t hold. The first argument will be contained in a message it prints.

abort

aborts the execution. i.e., exits the test’s execution with non-zero exit value. A stack trace will be printed to stderr.

bud/logging.rc

debug

prints a message to stderr as a DEBUG level log message.

info

prints a message to stderr as a INFO level log message.

error

prints a message to stderr as a ERROR level log message.

bud/json.rc

to_json_array

prints a JSON array composed of given arguments.

json_value_at

prints a JSON node specified by a path (second argument) in a given JSON node (first argument) json_has_key:: checks if an object node specified by a given path (third argument, default: ., which is root) in a given object node (first arguement) has a specified key (second argument). true will be printed, if it has. false, if not.

json_type_of

prints a type (object, array, string, number, boolean, and null) of a given JSON node (first argument). An error will be reported if it is not a malformed JSON string.

json_object_merge

merges two object nodes into one. If the same key appears, the second overrides the first.

json_array_append

prints a JSON array by appending an array (second argument) to the other (first argument).

Default Directory Layout

By default, commandunit assumes the compatible directory structure with maven based project[2].

Default Directory Layout

src/
  main/
  test/                                 (1)
    .commandunit/                       (2)
      jsonpp/
    scripts/
target/
  commandunit/
    report/                             (3)
    work/                               (4)
      scripts/
        core/
          {testcase-1}.json
          {testcase-1}.json++
          {testcase-1}.yaml++

1. A directory to store test scripts. (test script directory)

2. A directory to store commandunit 's configuration. (test project configuration directory)

3. A directory to which commandunit writes report. (report directory)

4. A directory to which commandunit stores processed test scripts. (work directory)

You can change the directory layout through commandunit 's options. Check the documentation.

Test Script Directory and Test Project Configuration Directory

By default, ./src/test is considered a directory that stores your test scripts. You can specify it by --test-srcdir option of the commandunit.

src/
  test/
    .commandunit/
      jsonpp/
    scripts/

Right under the test script directory, you can place a test project configuration directory.

A directory named .jsonpp directory under test project configuration directory will be added to JF_PATH environment variable. So that you can reference a file under it from $extends attribute.

Files in Report Directory

Following is the content of "report directory" after commandunit execution. You can specify a report directory by --test-reportdir option of commandunit and its default is target/commandunit/report.

Report Directory

target/
  commandunit/
    report/
      {testsuite-directory-name}/
        {testsuite-directory}/{testcase-1}.json-output/
        {testsuite-directory}/{testcase-2}.json-output/
      testreport.json
      testreport.adoc

For each of the {test-case-#} directory, following files are generated after an execution of when clause in a test script.

{testsuite-directory}/{testcase-#}.json-output/
  execution.txt
  exit_code.txt (1)
  precheck.txt
  script.txt    (2)
  stderr.txt    (3)
  stdout.txt    (4)

1. A file to store exit code from when clause.

2. A file to store the script generated by converting the content of when clause.

3. A file to store the content which 2. wrote to stderr.

4. A file to store the content which 2. wrote to stdout.

From those, testreport.json file, and then from it, testreport.adoc file is generated.

Test Report

After all tests are executed, testreport.json is generated and then testreport.adoc is generated from it.

Example 1. testreport.adoc example

]

To browse the rendered report, following tools will be useful.

• AsciidocFX[3]

• Intellij IDEA[4] + Asciidoc plugin[5]

In case you need to render it into an html file, you can use a command line tool called asciidoctor [6].

Work Directory

Under work directory, commandunit generates internal files that usually you are not interested in. You can specify the path of the directory by --test-workdir, and its default is target/commandunit/work.

Design of commandunit

The commandunit uses jq-front[1] to let users write tests. jq-front is a tool that allows you to use inheritances and node references (and more) in JSON files.

When it finds a file ends with .yaml++, the tool converts it to a JSON file first and the converted file will have a suffix .json++. Then the .json++ files will be converted into '.json' file using jq-front. (preprocess stage) So, familiarizing yourself with jq-front 's syntax will be very useful.

After this preprocessing step is finished, based on the content of the final JSON files, which should contain full-information to execute a test case, commandunit executes the test case. (run stage)

Then, from the files generated during the "run stage", commandunit generates a test report.

Check Design Concept and Design Detail pages of commandunit for more detail.

Installation Troubleshooting

The installer (install.sh) gives output like following:

$ ./install.sh
clean:    Removing target/commandunit/install
precheck: pass: <is_HOME_bin_in_PATH>
precheck: pass: <does_HOME_bin_exists>
precheck: pass: <is_yaml2json_installed>
precheck: pass: <is_jq_installed>
precheck: pass: <is_docker_installed>
precheck: pass: <docker_run_helloworld_works>
precheck: pass: <docker_run_mktemp_works>
precheck: pass: <is_bash_installed>
precheck: pass: <is_bash_modern_enough>
precheck: ----
precheck: FAILED CHECKS: 0
install:  INSTALLING commandunit...
install:  DONE
postcheck:pass: <installed_executable_is_found_by_which_command>
postcheck:pass: <docker_execution_exits_with_non_zero>
postcheck:pass: <failed_test_in_tap_report_is_one>
postcheck:pass: <testreport_json_exists>
postcheck:pass: <testreport_adoc_exists>
postcheck:pass: <testreport_json_num_failed_is_1>
postcheck:pass: <native_execution_exits_with_non_zero>
postcheck:pass: <failed_test_in_tap_report_is_one>
postcheck:pass: <testreport_json_exists>
postcheck:pass: <testreport_adoc_exists>
postcheck:pass: <testreport_json_num_failed_is_1>
postcheck:----
postcheck:FAILED CHECKS: 0
INSTALLATION SUCCEEDED

If the installation fails, please include the installer’s output in your ticket. Also, please attach an archive that contains files under target/commandunit/install (Create it by tar cvzf install-report.tar.gz target/commandunit/install).

To report a problem in the installer, please visit here.

References

• [1] jq-front project in github.org. jq-front: 2019

• [2] Apache Maven Project Apache Maven Project: 2002-2022

• [3] Asciidoc Book Editor based on JavaFX 18 Asciidoc Book Editor based on JavaFX 18: 2022

• [4] IntelliJ IDEA IntelliJ IDEA: 2000-2022

• [5] Asciidoctor Intellij Plugin Project Asciidoctor Intellij Plugin Project: 2022

• [6] Generate HTML from AsciiDoc Generate HTML from AsciiDoc: 2022