summaryrefslogtreecommitdiffstats
path: root/wiki/src/contribute/release_process/test/usage.mdwn
blob: cd75f23bb774e050b586922e96a9838b9e96ca21 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
[[!meta title="Running the automated test suite"]]

[[!toc levels=1]]

Basic usage
===========

Use the `run_test_suite` script found in the Tails source root to run
all automated Cucumber test features. See the [[setup
documentation|test/setup]] in case you don't have a testing
environment yet.

It's important to note that some features only depend on the Tails
sources, and some on the actual product of the sources, i.e. a Tails
ISO image. These features are tagged `@source` and `@product`,
respectively. The arguments passed to `run_test_suite` may only affect
one of these types of features and not the other.

A typical example run of a few `@product` features could be:

    ./run_test_suite --view --capture test-0.17.webm \
        --iso path/to/tails.iso \
        features/apt.feature features/erase_memory.feature

which will test only the `apt` and `erase_memory` features (if
no feature paths are given, all features in `features/cucumber` will
be tested) of the given ISO image `tails.iso` while showing the test
session in a VNC viewer (`--view`) and also capturing it into a video
called `test-0.17.web` (`--capture`). Similarly, to test a `@source`
feature, we'd simply run something like:

    ./run_test_suite features/build.feature

For full instructions, see its `--help`.

Note: since the test suite itself uses `virt-viewer` to interact with
the Tails guest you cannot use it as that will steal the session from
the test suite, leaving it with a blank screen.

Configuration
=============

The test suite can be configured in the following ways:

1. `run_test_suite` parameters, which takes precedence over

2. the local configuration file `features/config/local.yml`, which
takes precedence over

3. the default configuration file `features/config/defaults.yml`.

However, some values are treated as secrets and have no
defaults. These secrets are generally information about online sevices
to be used in certain features, like host, port and credentials --
stuff we don't want to make public. These must be set explicitly in
order for those features to run.

Here's a list of all non-secret key-value pairs that can be supported
by the local configuration file:

* `DEBUG`: Boolean value. If set to `true`, various debugging info
  will be printed. Defaults to `false`.

* `PAUSE_ON_FAIL`: Boolean value. If set to `true`, the test suite run
  is suspended on failure until ENTER is pressed. This is useful for
  investigating the state of the VM guest to see exactly why a test
  failed. Defaults to `false`.

* `SIKULI_RETRY_FINDFAILED`: Boolean value. If set to `true`, print a
  warning whenever Sikuli fails to find an image and allow *one* retry
  after pressing ENTER. This is useful for updating outdated images,
  or when extracting new images. Defaults to `false`.

* `TEMP_DIR`: String value. Directory where various temporary files
  are written during a test, e.g. VM snapshots and memory dumps,
  failure screenshots, pcap files and disk images. Defaults to
  `"/tmp/TailsToaster"`.