summaryrefslogtreecommitdiffstats
path: root/wiki/src/contribute/release_process/test/automated_tests.mdwn
blob: ff184f48dbd31d9bb45882899601660946b42762 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
[[!meta title="Automated test suite"]]

[[!toc levels=3]]

# Introduction

The automated test suite is essentially a glue between:

* [Sikuli](http://www.sikuli.org/) and
  [dogtail](https://fedorahosted.org/dogtail/), used for simulating
  user interaction,
* [libvirt](http://libvirt.org/), used for running a specific build of
  Tails in a virtual machine, and
* [cucumber](http://cukes.info/), for defining features and scenarios
  testing them using the above two components.

Its goal is to automate the development and release testing processes
with a Continuous Integration server.

A [[!tails_gitweb features/support/helpers/sikuli_helper.rb
desc="custom thin Ruby wrapper"]], using the
[rjb](http://rjb.rubyforge.org/) Ruby-Java bridge, gives us access to
the Sikuli programmatic interface from our step definitions.

## Setup and usage

See [[contribute/release_process/test/setup]] and [[contribute/release_process/test/usage]].

Core developers can also run it [[usage/on_lizard]].

## Features

With this tool, it is possible to:

  * Create, modify and destroy virtual machines that can run Tails from
    a variety of media (primarily DVD and USB drives).
  * Create different kinds of virtual storage (IDE, USB, DVD...),
    and either cold or hot plug/unplug them into/from the
    virtual machine.
  * Modify the virtual machine's hardware, e.g. its amount of RAM,
    its processor features (PAE), etc.
  * Simulate user interaction with the system under testing and check
    its state.
  * Run arbitrary shell commands inside the virtual machine.
  * Take screenshots from the display of the virtual machine, at
    particular events, like when a scenario fails. Complete test
    sessions can also be captured into a video.
  * Capture and analyze the network traffic of the system under testing.

## source vs. product cucumber features

Fundamentally speaking there are two types of tests:

* Tests that make sure that the Tails *sources* behave correctly *at
  build time* (example: `features/build.feature`); these ones are
  aptly tagged `@source`; and,
* Tests that make sure that the product built from Tails sources, i.e.
  a Tails ISO image, behaves correctly *at run time*; these ones are
  tagged `@product`.

The requirement of these are quite different; for instance, a
`@product` test must have access to a Tails ISO image to test, whereas a
`@source` test doesn't. Therefore their environments are setup
separately using our custom `BeforeFeature` hook, with the respective
tag.

# Implementation

## Running cucumber in the right environment

The `run_test_suite` script is a wrapper on top of `cucumber`, that
sets the correct environment up:

* It uses `Xvfb` so that the `DISPLAY` environment variable points to
  an unused X display (tip: use `Xvfb`) with screen size and color
  depth matching what the Sikuli images are optimized for (that is,
  1024x768 / 24-bit).
* It sets the `SIKULI_HOME` environment variable to the directory
  where sikuli script is installed for Java (`/usr/share/java` in
  Debian).
* It runs `unclutter` on `$DISPLAY` to prevent the mouse
  pointer from masking GUI elements looked for by Sikuli.
* It passes `--format ExtraHooks::Pretty` to `cucumber` calls, to get
  access to our custom `{After,Before}Feature` hooks.

## Remote shell

This started out as a hack, and while it has evolved it largely
remains so. A proper, reliable, established replacement would be
welcome, but seems unlikely given the requirements:

Requirements on the host (the remote shell client):

* can execute a command with blocking until command completion on the
  server, and get back return code, stdout and stderr separately.
* can spawn a command without blocking (except an ACK that the server
  has run the command, perhaps).
* usable by an unprivileged user

Requirements on the guest (the remote shell server):

* has to work without a network connection.
* should interfere minimally with the surrounding system (e.g. no
  firewall exceptions; actually we don't want any network traffic at
  all from it, but this kind of follows from the previous requirement
  any way)
* must start before Tails Greeter. Since that's the first point of
  user interaction in a Tails system (if we ignore the boot menu), it
  seems like a good place to be able to assume that the remote shell
  is running.

Scripts:

* [[!tails_gitweb config/chroot_local-includes/usr/local/lib/tails-autotest-remote-shell]]
* [[!tails_gitweb config/chroot_local-includes/lib/systemd/system/tails-autotest-remote-shell.service]]

# The art of writing new product test cases

Writing new `@source` features, scenarios or steps should be pretty
straightforward for anyone with experience with `cucumber`, but the
same can't be said about `@product` tests, so below we give some
pointers for the latter.

## Resources

In addition to the Sikuli, libvirt and Cucumber documentation linked
to in the introduction:

 * [ruby-libvirt API](http://libvirt.org/ruby/api/index.html)
 * [sikuli API](http://doc.sikuli.org/toc.html)
 * The `sniff` application, which is installed in Tails, is quite
   useful to navigate the GUI element hierarchy. However, `accerciser`
   (not installed) is even better, due to its ability to highlight
   elements. Another useful tool is `ipython` (not installed) with its
   TAB-completion.

## Writing new features and scenarios

First, one should have a good look at especially
`features/step_definitions/common_steps.rb` and the other features to
get a general idea of what already is possible. There is a good chance
there's already implementations for many steps necessary for reaching
the desired state right before when the stuff special to the intended
feature begins.

### Common steps example

In order to learn some basic step dependencies, and concepts and
features of the automated test suite used in features and scenarios,
let's walk through a few typical steps, in order:

    Given a computer

This is how each scenario (or background) should start. This step
destroys any residual VM from a previous scenario, and sets up a
completely fresh one, with all defaults. The defaults are defined in
`features/domains/default.xml`, but some highlights are:

* One virtual `x86_64` CPU with one core
* A reasonable amount of RAM
* ACPI, APIC and PAE enabled
* UTC harware clock
* A DVD drive loaded with the Tails from the ISO
* No other storage plugged
* USB 2.0 controller
* Ethernet interface, plugged into a network bridged with the host

However, most of the time we do not set up a computer from scratch
using this step, but restore from a snapshot (also called checkpoint)
using the one of the `Given Tails has booted ...` steps generated in
`features/step_definitions/snapshots.rb`. An example of such a step,
and indeed one of the most common ones, is:

    Given Tails has booted from DVD and logged in and the network is connected

These steps will actually run multiple steps, saving one or more
snapshots along the way. See the next section for details about this.

Returning back to what we'd do after the `Given a computer` step,
there's a number of steps that reconfigures the computer...

    And I create a 10 GiB disk named "some_disk"

The identifier (`some_disk`) is later used if we want to plug it or
otherwise act on it. Note that all media created this way are backed
by [[!wikipedia qcow2]] images, which grow only as they consume
capacity. All such media are destroyed after the feature ends.

This step does not necessarily have to be run this early, but it does
if we want to plug it as a non-removable drive...

    And I plug ide drive "some_disk"

Note that we can decide which type of drive `some_disk` is here. If we
pick a non-removable media, like in this case, it has to be done
before we start the virtual machine. Removable media, such as USB
drives, can be plugged into a live system at any later point.

    And the network is plugged

This plugs the network interface to the host-bridged network (which is
the default). There's also the "unplugged" version, which generally is
preferred for features that don't rely on the network (mostly so we
won't hammer the Tor network unnecessarily). These steps can also be
performed on a already running system under test.

    And I start the computer

This is where we actually boot the virtual machine.

    And the computer boots Tails

This step:

* verifies that we see the boot menu
* adds any boot options added via `I set Tails to boot with options ...`
* makes sure that Tails Greeter starts
* makes sure that the remote shell is up and running

    And I enable more Tails Greeter options

This is required for steps enabling Tails Greeter options, like the
next one.
Note that the "I set sudo password ..." step has to be run before the
other Tails Greeter option steps as it relies on keyboard navigation.

    And I set sudo password "asdf"

Beyond the obvious, after this step all steps requiring the
administrative password have access to it.

    And I log in to a new session
    And the Tails desktop is ready
    And I have a network connection
    And Tor is ready

All these should be pretty obvious. It could be mentioned that the
last two steps, like many others, depend on the remote shell to
be working.

    And all notifications have disappeared

The notifications can block GUI elements that we're looking for later
with Sikuli, so it's important that they are gone in essentially all
tests of GUI applications. If we have a network connection, so the
time syncing starts and shows its notifications, then this step should be
run after the previous step. Otherwise it always depends on the `GNOME
has started` step.

    And available upgrades have been checked

Since Tor is working, the check for upgrades will be run. We have to
wait for it to complete because that generally will break later if we
use snapshots.

### Snapshots

To speed up the test suite and get consistent results when setting up
state, we make heavy use of virtual machine snapshots. We encourage
contributors to read the snapshot definitions in
`features/step_definitions/snapshots.rb` carefully. We generate steps
from these descriptions, and they are created lazily on first use (and
then reused in subsequent instances, across features). Some things to
make note of:

* Snapshots may have parents, which means that they start by running
  the parent's step, generating its snapshot, recursively to a "root"
  snapshot without parent.

* Snapshots can be made "temporary". If the snapshot description's
  `:temporary` field is set to `true`, then the snapshot
  will be cleared after the feature it was created in finishes. This
  is a way to reduce the disk space needed for running the test suite,
  and is encouraged to use for features where a very perticular state
  is set up, that isn't reused in any other feature.

* Debugging snapshot creation is made a lot easier by enabling the
  `debug` formatter, which will print the steps as they are run.

### Scenarios involving the Internet

Each such new scenario should sniff network traffic, and check that
all Internet traffic has only flowed through Tor. See the `apt`
feature for a simple example of how to do it.

## Writing new steps

### The tools and some guidelines for their usage

In essense, the tools we have for interacting with the Tails instance
are:

* the VM helper class,
* Sikuli and dogtail, and
* the remote shell.

It should be fairly obvious when to use the VM helper class (stuff
relating to the virtual hardware), but there is some overlap between
Sikuli, dogtail and the remote shell.

Sikuli and dogtail:

* They should be used in all instances where we want to simulate user
  interaction with the Tails system. For instance, when we start an
  application we usually want to click our way through the GNOME
  applications menu to stay true to what an actual user would do.
* They come in handy when we want to verify some state pertaining
  to the GUI.
* In general we prefer dogtail, since it is more precise given its
  direct access to GUI elements and their state, and Sikuli's images
  carry a quite heavy maintenance burden. However, Dogtail is not
  possible to use before GNOME has started, so we rely on Sikuli for
  anything before or after that. Also, for some applications it's
  simply hard to "navigate" to the right element in the hierarchy of
  elements of some GUI, due to ambiguity and/or lack of labels.

The remote shell:

* Useful for testing non-GUI state.
* Useful for reaching some state required before we test stuff
  depending on user interaction (which should use Sikuli!).
* It is acceptable (but not encouraged) to use the remote shell for
  simulating user terminal interaction when we need to analyze the
  commands output. This is because of Sikuli's OCR capabilities are
  poor, and cannot be depended on.

# Limitations and issues

These things are good to know when developing new features, scenarios
or steps.

## The remote shell

### Different behaviour compared to "real" shell

The remote shells have a few surprises in store:

* `pgrep -f` detects itself, which can have potentially serious
  consequences if not dealt with.
* Minor groups are not set, so e.g. the `groups` command may not do
  what you expect, but `groups $USER` does.

These are the currently known oddities of the remote shell, but there
may be more, so beware, and make sure to verify that any commands sent
through it does what you want them to do.

### Remote shell instability

Although very rare, the remote shell can get into a state where it
stops responding, resulting in the test suite waiting for a response
forever.

## Host-to-guest filesystem shares are incompatibile with snapshots

Filesystem shares cannot (due to QEMU limitations) be added to an
active VM, and cannot (due to QEMU limitations) be active
(i.e. mounted) during a snapshot save. For this reason, don't use
filesystem shares in combination with snapshots. For more
information, see [[!tails_ticket 5571]].

On a more practical note, you *can* add a filesystem share if you
restore a snapshot and then power off the computer, which still is
worth it when there's a big setup cost, e.g. when Tails is running
from USB with persistence enabled. So something like this is valid,
for example:

    Given Tails has booted without network from a USB drive with a persistent partition and stopped at Tails Greeter's login screen
    And I shutdown Tails and wait for the computer to power off
    And I setup some filesystem share ...
    And I start Tails from USB drive "current" with network unplugged and I login with persistence enabled

## Plugging SATA drives

When creating a disk (at least when backed by a `raw` image) via the
storage helper, and then plugging it to a Tails instance as SATA
drive, GNOME will report that the drive is failing when inside Tails,
and indeed several SMART tests fail. For now, plug hard disks as IDE
only (or USB, of course).

## Passing state between steps in snapshots

When creating snapshots, anything stored in a variable in any of those
steps will only be available in subsequent steps in that scenario, not
in other scenarios restoring from that snapshot. The exception is when
global variables are used, which is an acceptable workaround, but it
requires minute control to get right, and is hard to follow. Please
try to avoid this until [[!tails_ticket 5847]] is solved.

## Disabling scenarios that are known to fail

When a scenario is broken for an extended period of time (e.g. when
rebasing Tails on a new Debian version, which usually breaks tons of
stuff) the current method of temporarily disabling affected tests is
simply to remove them in Tails' Git, making sure that we won't forget
about them.

Note that such removals should be isolated per commit so that they are
easy to revert when whatever was broken is fixed. Hence, such a commit
could either:

* remove a single scenario if something unique to that scenario is
  broken;

* remove multiple scenarios possibly spanning multiple features, if
  they're broken for the same reason, e.g. a step they all use is
  broken;

* remove a complete feature, if the complete feature is broken for the
  same reason.

Each such commit *must* have a ticket created, referencing the commit
to revert, have an assignee and an appropriate milestone set so they
are not forgotten. Bonus point: push a branch that reverts the commit,
and then set it as the ticket's _Feature Branch_.

(Once [[!tails_ticket 10198]] has hit Cucumber we can solve this,
i.e. [[!tails_ticket 7233]], in a much better way.)