summaryrefslogtreecommitdiffstats
path: root/wiki/src/contribute/release_process/test/usage.mdwn
blob: 813d67770f45cbb06d65fb14f3310781250299b3 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
[[!meta title="Running the automated test suite"]]

[[!toc levels=2]]

Basic usage
===========

Use the `run_test_suite` script found in the Tails source root to run
all automated Cucumber test features. See the [[setup
documentation|test/setup]] in case you don't have a testing
environment yet.

It's important to note that some features only depend on the Tails
sources, and some on the actual product of the sources, i.e. a Tails
ISO image. These features are tagged `@source` and `@product`,
respectively. The arguments passed to `run_test_suite` may only affect
one of these types of features and not the other.

A typical example run of a few `@product` features could be:

    ./run_test_suite --view --capture test-0.17.webm \
        --iso path/to/tails.iso \
        features/apt.feature features/erase_memory.feature

which will test only the `apt` and `erase_memory` features (if
no feature paths are given, all features in `features/cucumber` will
be tested) of the given ISO image `tails.iso` while showing the test
session in a VNC viewer (`--view`) and also capturing it into a video
called `test-0.17.web` (`--capture`). Similarly, to test a `@source`
feature, we'd simply run something like:

    ./run_test_suite features/build.feature

For full instructions, see its `--help`.

Note: since the test suite itself uses `virt-viewer` to interact with
the Tails guest you cannot use it as that will steal the session from
the test suite, leaving it with a blank screen.

Configuration
=============

The test suite can be configured in the following ways:

1. `run_test_suite` parameters, which takes precedence over

2. the local configuration file `features/config/local.yml`, which
takes precedence over

3. the default configuration file `features/config/defaults.yml`.

However, some values are treated as secrets and have no
defaults. These secrets are generally information about online sevices
to be used in certain features, like host, port and credentials --
stuff we don't want to make public. These must be set explicitly in
order for those features to run.

## Non-secret configuration

Here's a list of all non-secret key-value pairs that can be supported
by the local configuration file:

* `DEBUG`: Boolean value. If set to `true`, various debugging info
  will be printed. Defaults to `false`.

* `PAUSE_ON_FAIL`: Boolean value. If set to `true`, the test suite run
  is suspended on failure until ENTER is pressed. This is useful for
  investigating the state of the VM guest to see exactly why a test
  failed. Defaults to `false`.

* `SIKULI_RETRY_FINDFAILED`: Boolean value. If set to `true`, print a
  warning whenever Sikuli fails to find an image and allow *one* retry
  after pressing ENTER. This is useful for updating outdated images,
  or when extracting new images. Defaults to `false`.

* `TEMP_DIR`: String value. Directory where various temporary files
  are written during a test, e.g. VM snapshots and memory dumps,
  failure screenshots, pcap files and disk images. Defaults to
  `"/tmp/TailsToaster"`.

## "Secret" configuration

This section describes the formats for all secret configurations that
must be configured in the local configuration file for certain
features or scenarios to work. If any of these are omitted, parts of
the test suite will fail.

### Tor pluggable transports

The format is:

    Tor:
      Transports:
        $TYPE:
          - ipv4_address: "1.2.3.4"
            ipv4_port: 443
            fingerprint: "01234567890abcdef01234567890abcdef012345"
            extra:
          - ipv4_address: "5.6.7.8"
          [...]
        $ANOTHER_TYPE:
          - ipv4_address: "1.2.3.4"
          [...]

where the type `$TYPE` (and `$ANOTHER_TYPE`) should be something like
`obfs4` or `bridge` (the first type) or whatever Tor calls them. Both
`fingerprint` and `extra` are optional and can be left empty (or
skipped completely), but e.g. `extra` is necessary for `obfs4` type
bridges, for the `cert=... iat-mode=...` stuff, and the same for
`scramblesuite`'s `password=...`.

This setting is required for `tor_bridges.feature` (requires types
`bridge`, `obfs2`, `obfs3` and `obfs4`) and `time_syncing.feature`
(requires type `bridge` only).