TESTS

E2E

Central Script

The central script is e2e.sh. The syntax for calling it can be obtained with e2e.sh -h

The script is integrated into the makefile as targets named "e2e" and "dry".

make e2e is the central command in the Gitlab pipeline to test the artifacts. It must succeed for a deployment.

Test Suites

A complete test consists of a colletion of test suites.

There are test suits for basic tests, regressions, vocabularies and edge cases.

Test Suite Structure

A test suite is bundled in a directory named Suite*.
Optionally there can be a file suite.inc per test suite. This shell
script is sourced in, allowing you to set environment variables. These
environment variables can be used to configure the behavior of e2e.sh.

• SUITE_CHECKKSN=0|1 ..... controls whether to compare the parser output
  with a given .ksn file. The default value is 1.
• SUITE_BOOTKS=0|1 ....... controls whether to include the boot.ks file.
  The default value is 0.
• SUITE_WILDCARD=0|1 ..... controls whether to accept bash wildcards in
  test*.out files. The default value is 0.
• SUITE_VOCABULARY=voc ... controls whether to source in the vocabulary
  called voc. voc can be a comma-separated list of names.
  The default value is empty.

Test Cases

Each test suite consists of multiple test cases. e2e.sh always runs
all test cases within a given test suite directory.

Each test case consists of two or three files:

• test*.ks: contains the input for Kryptostack.
• test*.ksn (optional): contains the expected output of the parser pass
  (if SUITE_CHECK_KSN=1).
• test*.out: contains the expected output of the interpreter.

Comments in Test Cases

The first line of each .ks file should contain a comment. This comment
is used for progress reporting during an e2e.sh run.

Test Execution

1. Iterating Through Test Suites: First, e2e.sh iterates through all
   Suite* test suite directories provided on the command line.
2. Iterating Through Test Cases: Within each test suite, e2e.sh
   then iterates through all individual test cases.
3. Executing Test Cases:
   • The test*.ks file is executed as a Kryptostack program by the parser.
   • The parser's output is optionally compared to the test*.ksn file (if it exists).
   • Finally, the interpreter's output is compared to the test*.out file.

BOOST Unit Tests

make generate a utest executable.

It used to test the system on a class and function level.

With build type PROFILE utest generates additional data for the lcov
coverage statistics.

Following Usage Variants the "Static library usage variant" is applied.

Coverage Tests

The central script is Tools/dolcov.sh.
The syntax for calling it can be obtained with Tools/dolcov.sh -h

The script is integrated into the makefile as target named "lcov".

Coverage tests runs with the build type PROFILE.

The command make lcov generates profile data by running

• e2e.sh checks
• BOOST unit tests
• additional command line checks

and using lcov to generate HTML documents in ./public/lcov.

A manual inspection of the HTML output is needed to assess the quality of the tests.

Following exclusion markers for lcov are in use. See man page of geninfo.

• LCOV_EXCL_LINE - Lines containing this marker will be excluded.
• LCOV_EXCL_START - Marks the beginning of an excluded section.
  The current line is part of this section.
• LCOV_EXCL_STOP - Marks the end of an excluded section. The current line
  is not part of this section.

Examples

A test run of all examples is part of the test stage in the pipelines.
One can execute this task manually with make examples.
See EXAMPLES