[[!meta title="Running the automated test suite"]]
Note that all traffic that normally would be sent through the Tor
network is sent directly from the system running the automated test
suite to the destinations! In other words, nothing is torified, and
some tests contact hosts associated with Tails, Tor, Riseup etc. It
will, for instance, be obvious to your ISP that you are running the
Tails automated test suite.
Use the `run_test_suite` script found in the Tails source root to run
all automated Cucumber test features. See the [[setup
documentation|test/setup]] in case you don't have a testing
environment yet. Note that the full Tails source tree must be readable
by the user running the test suite.
It's important to note that some features only depend on the Tails
sources, and some on the actual product of the sources, i.e. a Tails
ISO image. These features are tagged `@source` and `@product`,
respectively. The arguments passed to `run_test_suite` may only affect
one of these types of features and not the other.
A typical example run of a few `@product` features could be:
./run_test_suite --view --capture test-0.17.webm \
--iso path/to/tails.iso \
which will test only the `apt` and `erase_memory` features (if
no feature paths are given, all features in `features/cucumber` will
be tested) of the given ISO image `tails.iso` while showing the test
session in a VNC viewer (`--view`) and also capturing it into a video
called `test-0.17.web` (`--capture`). Similarly, to test a `@source`
feature, we'd simply run something like:
Actually, `run_test_suite` is just a wrapper around `cucumber`, so any
`cucumber` option can be passed too, although after an `--` so they
are not confused with the wrapper's options. For instance:
./run_test_suite ... -- --format debug features/apt.feature
will enable the `debug` formatter, which in Tails' Cucumber setup will
enable debugging information to be printed (which is *very* useful
when debugging or developing for the test suite) unlike in vanilla
Cucumber, where it's used for debugging the formatting subsystem.
For full instructions, see its `--help`.
Note: since the test suite itself uses `virt-viewer` to interact with
the Tails guest you cannot use it as that will steal the session from
the test suite, leaving it with a blank screen.
The test suite can be configured in the following ways:
1. `run_test_suite` parameters, which takes precedence over
2. the local configuration file `features/config/local.yml`, which
takes precedence over
2. the local configuration files in `features/config/*.d` (with
internal precedence according to lexical order), which takes
4. the default configuration file `features/config/defaults.yml`.
However, some values are treated as secrets and have no
defaults. These secrets are generally information about online sevices
to be used in certain features, like host, port and credentials --
stuff we don't want to make public. These must be set explicitly in
order for those features to run.
A Git repository, shared among a bunch of core Tails contributors,
includes _some_ of the needed secrets (more specifically, those that
can be use by concurrent test suite runs):
git clone firstname.lastname@example.org:test-suite-shared-secrets \
## Non-secret configuration
Here's a list of all non-secret key-value pairs that can be supported
by the local configuration file:
* `CAPTURE`: Captures failed scenarios into videos stored in the
temporary directory (see `TMPDIR` below) using x264
encoding. Defaults to `false`.
* `CAPTURE_ALL`: Keep videos for all scenarios, including those that
succeed (implies `CAPTURE`). Defaults to `false`.
* `MAX_NEW_TOR_CIRCUIT_RETRIES`: Integer. Upon failure, some test steps may be
run again after requesting that connections are made using new Tor circuits. This
configuration variable limits how many times forcing a circuit will be
attempted. Defaults to `10`.
* `INTERACTIVE_DEBUGGING`: Boolean value. If set to `true`, the test
suite run is suspended on failure until ENTER is pressed, and an
interactive Ruby shell (pry) is available. This is useful for
investigating the state of Cucumber's world and the VM guest to see
exactly why a test failed. Defaults to `false`.
* `SIKULI_RETRY_FINDFAILED`: Boolean value. If set to `true`, print a
warning whenever Sikuli fails to find an image and allow *one* retry
after pressing ENTER. This is useful for updating outdated images,
or when extracting new images. Defaults to `false`.
* `TMPDIR`: String value. Directory where various temporary files
are written during a test, e.g. VM snapshots and memory dumps,
failure screenshots, pcap files and disk images. Defaults to
* `NOTIFY_USER_COMMAND`: String value. This arbitrary shell command
will be executed whenever `pause()` is called, e.g. on test failure
when `INTERACTIVE_DEBUGGING` (`--interactive-debugging`) is
enabled. This is pretty useful when multitasking with long test
suite runs, so you immediately are notified when a test fails (or
when you reached a temporary `pause()` breakpoint).
## "Secret" configuration
This section describes the formats for all secret configurations that
must be configured in the local configuration file for certain
features or scenarios to work. If any of these are omitted, parts of
the test suite will fail.
These secrets are required for some scenarios in
`pidgin.feature`. Here's an example which explains the format:
Note that the fields used in the above example show the *mandatory*
The XMPP account described by `Tails_account` (to be used in Tails'
Pidgin) and `Friend_account` (run via a bot from the tester host) must
be subscribed to each other but to no other XMPP account. Also, for the
`Friend_account`, it's important that the `otr_key`'s `name` field is
the same as `username`, and that the `protocol` field is `xmpp`.
If a "Connect Server" is needed for any of the accounts, it can be set
in the *optional* `connect_server` field.
In case the `Tails_account`'s conference server doesn't allow creating
arbitrary chat rooms, a specific one that is known to work can be set
in the *optional* `chat_room` field. It should still be a room with a
strange name that is highly likely to always be empty; otherwise the
test will fail.
XMPP services known to work well both for `Tails_account` and
* riseup.net (use `connect_server: xmpp.riseup.net`)
* jabber.org (doesn't allow creating arbitrary chatrooms, so setting
`chat_room` may be needed)
These settings are required for `ssh.feature`. The format is:
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
public_key: "ssh-rsa AAAAB3NzaC1yc2EA..."
where `$TYPE` is `SSH` or `SFTP`. Secrets must be specified for both `SSH` and
`SFTP`. If `port` is not specified, `22`will be used.
The SSH test expects the remote system to have a default `bash` shell prompt.
These settings are required for `icedove.feature`. The format is:
The email account's inbox must contain at least one email at all times.