summaryrefslogtreecommitdiffstats
path: root/wiki/src/contribute/release_process/test.mdwn
blob: 46373fae9bc32dc8d2172d7c87cfcdde4423a8a7 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
[[!meta title="Manual test suite"]]

[[!toc levels=1]]

Some [[test results]] that might be useful to keep are saved.

<div class="caution">
Read this document from the branch used to prepare the release.
</div>

# Changes

Keeping an eye on the changes between released versions is one of the
many safeguards against releasing crap.

## Source


Compare the to-be-released source code with previous version's one e.g.:

Boot the candidate image and find the commit it was build from with the
`tails-version` command.

Then, from the source tree, see the diff:

	git diff --find-renames <old image commit>..<candidate image commit>

e.g. `git diff --find-renames 334e1c485a3a79be9fff899d4dc9d2db89cdc9e1..cfbde80925fdd0af008f10bc90c8a91a578c58e3`

## Result

Compare the list of bundled packages and versions with the one shipped last
time. `.packages` are usually attached to the email announcing the image is ready.

	/usr/bin/diff -u \
	    wiki/src/torrents/files/tails-amd64-3.1.packages \
	    tails-amd64-3.2.packages \
	    | wdiff --diff-input  --terminal

Check the output for:

- new packages that may cause harm or make the images unnecessarily
  big
- packages that could be erroneously removed
- new versions of software we might not have audited yet (including:
  does the combination of our configuration with software X version
  Y+1 achieve the same wished results as with software X version Y?)

## Image size

Check the images size has not changed much since the last release.

In a directory with many Tails ISO and USB images:

    find \( -iname "tails*.iso" -o -iname "tails*.img" \) \
	     -type f -exec ls -l --block-size=M '{}' \; | sort -rhk 5

<a id="reproducibility-final-check"></a>

# Reproducibility

This section can **not** be done by the RM.

1. Download the ISO and USB images.

2. Clear-sign the hashes of all products using your OpenPGP key
   and gzip the output (otherwise the signed text could be mangled
   at some point in the email chain):

        DEST_DIR=$(mktemp -d)
        sha512sum *.iso *.img \
          | gpg --clear-sign \
          | gzip \
          > "$DEST_DIR/TR-bits.gz"

3. Locate the file generated by the command above. To display its location,
   execute the following command:

        echo "$DEST_DIR/TR-bits.gz"

4. Send the aforementioned generated file as an attachment
   to the _Trusted Reproducer_, whose name is on the
   [release calendar](https://tails.boum.org/contribute/calendar/).

5. If the _Trusted Reproducer_ is around, ask them:

    - if they've received your email
    - if they could successfully decompress the attachment
    - if they could check the inline signature of the attachment once decompressed
    - if the signed text contains all the information they need

# Automated test suite

Our long term goal is to eliminate the manual test suite (except the
parts which require real hardware) and have the automated test suite
run all our tests. Its design, and how to write new tests, are
documented on a [[dedicated page|test/automated_tests]].

## Running the automated test suite

See [[test/setup]] and [[test/usage]].

Do point `--old-iso` to the ISO image of the previous stable release.

## Automated test suite migration progress

The manual test suite below either contains tests that cannot be
automated, has no automated test implemented yet, or has a test
implemented, but it either hasn't been reviewed, had a confirmed pass
by someone other than the test author, or has issues. The latter is
tracked by tickets prefixed with `todo/test_suite:`.

# Tor Browser

## Miscellaneous functionality

* Test if _uBlock_ works:
  - The _uBlock_ icon must be visible.
  - Visit a website that normally displays ads, such as
    <https://www.nytimes.com/>. The ads should not be displayed and
    the uBlock icon should display a strictly positive number of
    blocked elements.

## Security and fingerprinting

* Run the [tests the Tor Browser folks
  use](https://trac.torproject.org/projects/tor/wiki/doc/build/BuildSignoff#TestPagestoUse)
  and compare to the last released version of Tails. Results should
  not be worse.
  (automate: [[!tails_ticket 10260]])
  - For the "evercookie" test to work, you may have to disable
    _uBlock_ on its web page.
* Compare the fingerprint Tor Browser in Tails with the fingerprint of
  the same version of Tor Browser (running on Linux outside of Tails), using at least
  <https://panopticlick.eff.org/> (automate: [[!tails_ticket 10262]]).
  Click "Show full results for fingerprinting" to see the details
  we're interested in.
  - The exposed User-Agent should match the latest Tor Browser's one.
  - Ignore the result of the "blocking tracking ads" and "blocking
    invisible trackers" tests, which seem unreliable (we've seen
    different results for the very same version of Tor Browser).
  - Update the *Browser fingerprint* section of the
    [[known issues|support/known_issues]] if needed.
* WebRTC should be disabled: (automate: [[!tails_ticket 10264]])
  - In `about:config` check that `media.peerconnection.enabled` is set to
    `false`.
  - <http://mozilla.github.io/webrtc-landing/>, especially the `getUserMedia`
    test. It's expected that the audio test works if you agree to share a
    microphone with the remote website; anything else should fail.
  - <http://net.ipcalf.com/> should display _literally_
    `ifconfig | grep inet | grep -v inet6 | cut -d" " -f2 | tail -n1`
* The content of `/etc/default/htpdate.user-agent` should match should produce
  the User-Agent used in the Tor Browser. (automate: [[!tails_ticket 10268]])

<a id="Thunderbird"></a>

# Thunderbird

* Check mail over IMAP using:
  - a hidden service IMAP server (e.g. Riseup, zsolxunfmbfuq7wf.onion on port 993 with SSL).
* Check mail over POP using:
  - a hidden service POP server (see above, on port 995 with SSL).
* Send an email using:
  - a hidden service SMTP server (see above, on port 465 with SSL).

* Check that the profile works and is torified:
  1. Send an email using Thunderbird and a non-anonymizing SMTP relay (a
     SMTP relay that writes the IP address of the client it is
     relaying email for in the Received header).
  1. Then check that email's headers once received, especially the
     `Received:` ones.
* Also check that the EHLO/HELO SMTP message is not leaking anything
  at the application level:
  1. Start Thunderbird using the GNOME Applications menu.
  2. Disable SSL/TLS for SMTP in Thunderbird (so take precautions for not
     leaking your password in plaintext by either changing it
     temporarily or using a disposable account). Or better, configure
     StartTLS, since it will send two EHLO/HELO: one before TLS is
     initiated; one after. The assumption here is that Thunderbird will
     send the same both times.
  3. Run `sudo tcpdump -n -i lo -w dump` while sending an email to
     capture the packets before Tor encrypts it, then close
     tcpdump. Note that the packet containing EHLO/HELO will be sent
     really early, so even if the email failed (e.g. because the mail
     server doesn't support plaintext SMTP on port 587) we are ok.
  4. Check the dump for the HELO/EHLO message and
     verify that it only contains `127.0.0.1`:
     `sudo tcpdump -A -r dump | grep EHLO`

# Tor

* The version of Tor should be the latest stable one, which is the highest version number
  before alpha releases on <http://deb.torproject.org/torproject.org/pool/main/t/tor/>. (automate:
  [[!tails_ticket 10259]])

# WhisperBack

* I should be able to send a bug report with WhisperBack.
* When we receive this bug report on the tails-bugs mailing list,
  Schleuder tells us that it was sent encrypted.

# Root access control

* Check you cannot login as root with `/bin/su` neither with the `amnesia` password nor
  with the `live` one. (automate: [[!tails_ticket 10274]])

# Virtualization support

* Test that Tails starts and the browser launches in VirtualBox.

# APT (automate: [[!tails_ticket 8164 desc="#8164"]])

     grep -r jenw7xbd6tf7vfhp.onion /etc/apt/sources.list*

* Make sure the Tails repository suite in matching the release tag (for example
  the release version number) is in APT sources.
* Make sure the Tails repository unversioned suites (e.g. `testing`,
  `stable` and `devel`) are *not* in APT sources.

<a id="incremental-upgrades"></a>

# Incremental upgrades

* List the versions from which an upgrade path to this one is described.
  On the `master` branch:

      git grep -E -l "  version: '?0.23'?" \
          wiki/src/upgrade/v1/Tails/*/*/test/upgrades.yml

  … replacing "0.23" with the version you are testing.

* For each description file, open it and verify if it allows incremental upgrade
  or only full upgrade.

* For each previous version from which an upgrade path is described, install it
  and try to upgrade:
  * For every incremental upgrade path: make sure the resulting updated
    system "works fine" (boots, pretends to be the correct version,
    and the following components work fine: Tor, Tor Browser, Unsafe Browser).
  * For upgrade paths that only propose a full upgrade: make sure the
    user is told to do a manual upgrade.

  If:

  * the update-description files have been published on the
    *test* channel already (see <https://tails.boum.org/upgrade/v1/Tails/>)
  * and the IUK has been published already (see
    <https://archive.torproject.org/amnesia.boum.org/tails/alpha/>
    and <https://archive.torproject.org/amnesia.boum.org/tails/stable/>):

  then:

        sudo sh -c 'sed -i /^TAILS_CHANNEL=/d /etc/os-release &&
                    echo TAILS_CHANNEL=\"test\" >> /etc/os-release' && \
        tails-upgrade-frontend-wrapper --no-wait

  Else, use a local test setup:

  * A web server on the LAN.
  * A copy of `wiki/src/upgrade` from the `stable` or `testing` branch,
    for example in `/var/www/tails/upgrade/v1/Tails/3.14~rc2/amd64/stable/updates.yml`
  * A copy of the `iuk` directory of our HTTP mirrors,
    for example in `/var/www/tails/stable/iuk/Tails_amd64_3.14-rc2_to_3.14.iuk`.

    To synchronize your local copy:

        torsocks rsync -rt --progress --delete mirrors.rsync.tails.boum.org::amnesia-archive/tails/stable/iuk/ /var/www/tails/stable/iuk/

  * Patch `/etc/hosts` in Tails to point to your web server:

        echo "192.168.1.4    dl.amnesia.boum.org" | sudo tee --append /etc/hosts

  * Patch sudo configuration to allow passing arbitrary arguments to
    `tails-upgrade-frontend`:

        sudo sed -i \
            -e 's,/usr/bin/tails-upgrade-frontend ""$,/usr/bin/tails-upgrade-frontend,' \
            /etc/sudoers.d/zzz_upgrade

  * Run the upgrader. It must be called, from inside the system to upgrade,
    with every needed option to use the local web server rather than the
    online one, for example:

        DISABLE_PROXY=1 SSL_NO_VERIFY=1 \
        tails-upgrade-frontend-wrapper --override-baseurl \
        http://192.168.1.4/tails

# Unsafe Web Browser

* Browsing (by IP) a FTP server on the LAN should be possible. (automate: [[!tails_ticket 10252]])

# Tails Verification

* The goal is to check that *Tails Verification* works in *Tor
  Browser* in the version of Tails we are testing here. *Tails
  Verification* only supports verifying the current release so for
  example, when doing tests for the Tails 3.13 release, we use it in
  the tentative Tails 3.13 to verify the Tails 3.12 ISO and USB images.

  1. Start the Tails that you are testing.

  2. Start *Tor Browser*.

  3. Visit <https://tails.boum.org/install/download/>.

  4. If you already have the ISO image for the last, already released
     version of Tails, i.e. the one that's advertised in the sidebar
     of our production website, then click "*I already downloaded Tails $N*".
     Otherwise follow the instructions on that web page to download
     the ISO image.

  5. Install *Tails Verification*.

  6. Click the **Verify Tails $N** button.

  7. The verification should be successful.

  8. Repeat for the USB image.

# Real (non-VM) hardware

`[can't-automate]`

* Boot on bare-metal from USB. Measure the boot time (from the
  syslinux menu until the GNOME desktop is ready -- quickly press
  ENTER in the Greeter) and compare with the boot time of the previous
  Tails version. The new one should not be significantly slower to
  start.
* Boot on bare-metal from DVD. Measure the boot time (from the
  syslinux menu until the GNOME desktop is ready -- quickly press
  ENTER in the Greeter) and compare with the boot time of the previous
  Tails version. The new one should not be significantly slower to
  start (for release candidates we do not always update the squashfs
  sort file, so then it might be ok if somewhat slower).

# Documentation

* The "Tails documentation" desktop launcher should open the
  [[doc]] page (automate: [[!tails_ticket 8788]]):
  - in one language to which the website is translated
  - in one language to which the website is not translated (=> English)
* Browse around in the documentation shipped in the image. Internal
  links should be fine. (automate: [[!tails_ticket 10254]])

# Internationalization

Boot and check basic functionality is working for these (language,
region) tuples:

 - Arabic - Egypt
 - Chinese - China
 - Deutsch - Deutschland
 - English - USA
 - Español - España
 - Français - France
 - Italiano - Italia
 - Persian - Iran
 - Português - Brasil
 - Russian - Russian Federation
 - Tiếng Việt - Vietnam

You *really* have to reboot between each language.

* The chosen keyboard layout must be applied. (automate: [[!tails_ticket 10261]])
* The screen keyboard must (automate: [[!tails_ticket 10263]]):
  - work in Tor Browser when activated after the browser has started;
  - work in Thunderbird when activated after Thunderbird has started;
  - be auto-configured to use the same keyboard layout as the
    X session (known exceptions: Chinese, Persian, Português,
    Russian, Tiếng Việt).
* In the Tor Browser:
  - DuckDuckGo must be the default, pre-selected search plugin. (automate: [[!tails_ticket 10265]])

## Spellchecking

To see which among the supported locales there should be no
spellchecker, run this in a Tails Git checkout of the commit the
release under testing was built from:

    git grep NO_SPELLCHECKER_LOCALES= config/chroot_local-hooks/11-localize_browser

Then do the follow in the same Tor Browser session running in the
`en_US.UTF-8` locale (or whatever locale you are most comfortable
identifying other language names in):

* Check that the expected languages are listed in the list of languages for
  spell checking. (automate: [[!tails_ticket 10269]])
  - Visit <https://translate.google.com/>.
  - Right-click and choose "Check spelling".
  - Right-click and check the list of available languages.
* For a few languages, check the spell checking:
  - Type something in the textarea.
  - Right-click and select a language.
  - Verify that the spelling suggestion are from that language. (automate: [[!tails_ticket 10271]])

# Misc

* Check that all seems well during init: (automate: [[!tails_ticket 10277]])
  - `systemctl --failed --all` should say `0 loaded units listed`
  - the output of `journalctl` should seem OK.