summaryrefslogtreecommitdiffstats
path: root/wiki/src/contribute/release_process/test.mdwn
blob: d145772dcb66d4c2d0b703f210d8d8df1e4c18a5 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
[[!meta title="Manual test suite"]]

[[!toc levels=1]]

Some [[test results]] that might be useful to keep are saved.

<div class="caution">
Read this document from the branch used to prepare the release.
</div>

# Changes

Keeping an eye on the changes between released versions is one of the
many safeguards against releasing crap.

## Source


Compare the to-be-released source code with previous version's one e.g.:

Boot the candidate ISO and find the commit it was build from with the
`tails-version` command.

Then, from the source tree, see the diff:

	git diff --find-renames <old ISO commit>..<ISO commit>

e.g. `git diff --find-renames 334e1c485a3a79be9fff899d4dc9d2db89cdc9e1..cfbde80925fdd0af008f10bc90c8a91a578c58e3`

## Result

Compare the list of bundled packages and versions with the one shipped last
time. `.packages` are usually attached to the email announcing the ISO is ready.

	/usr/bin/diff -u \
	    wiki/src/torrents/files/tails-i386-1.3.1.packages \
	    tails-i386-1.3.2.packages \
	    | wdiff --diff-input  --terminal

Check the output for:

- new packages that may cause harm or make the images unnecessarily
  big
- packages that could be erroneously removed
- new versions of software we might not have audited yet (including:
  does the combination of our configuration with software X version
  Y+1 achieve the same wished results as with software X version Y?)

## Image size

Check the image size has not changed much since the last release.

In a directory with many Tails ISO images:

    find -iname "tails*.iso" -exec ls -lh '{}' \; | sort -rhk 5

# Automated test suite

Our long term goal is to eliminate the manual test suite (except the
parts which require real hardware) and have the automated test suite
run all our tests. It's design, and how to write new tests, are
documented on a [[dedicated page|test/automated_tests]].

## Running the automated test suite

See [[test/setup]] and [[test/usage]].

Do point `--old-iso` to the ISO of the previous stable release.

## Automated test suite migration progress

The manual test suite below either contains tests that cannot be
automated, has no automated test implemented yet, or has a test
implemented, but it either hasn't been reviewed, had a confirmed pass
by someone other than the test author, or has issues. The latter is
tracked by tickets prefixed with `todo/test_suite:`.

# Tor Browser

## Security and fingerprinting

* Run the [tests the TBB folks
  use](https://trac.torproject.org/projects/tor/wiki/doc/build/BuildSignoff#TestPagestoUse).
* Compare the fingerprint of Tails and the latest TBB using at least
  <https://panopticlick.eff.org/>
  - The exposed User-Agent should match the latest TBB's one.
  - Update the [[fingerprint section|support/known_issues#fingerprint]] of the
    known issues page if needed.
* WebRTC should be disabled:
  - In `about:config` check that `media.peerconnection.enabled` is set to
    `false`.
  - <http://mozilla.github.io/webrtc-landing/>, especially the `getUserMedia`
    test. It's expected that the audio test works if you agree to share a
    microphone with the remote website; anything else should fail.
  - <http://net.ipcalf.com/> should display
    `ifconfig | grep inet | grep -v inet6 | cut -d" " -f2 | tail -n1`
* One should be able to switch identities from the web browser.
* Running `getTorBrowserUserAgent` should produce the User-Agent set by the
  installed version of Torbutton, and used in the Tor Browser.

# Tor

* The version of Tor should be the latest stable one, which is the highest version number
  before alpha releases on <http://deb.torproject.org/torproject.org/pool/main/t/tor/>.

# Claws

* Check mail over IMAP using:
  - a "clearnet" IMAP server.
  - a hidden service IMAP server (e.g. Riseup, zsolxunfmbfuq7wf.onion
    with SSL).
* Send an email using:
  - a "clearnet" SMTP server.
  - a hidden service SMTP server (see above).
* Check that the profile works and is torified:
  1. Send an email using Claws and a non-anonymizing SMTP relay (a
     SMTP relay that writes the IP address of the client it is
     relaying email for in the Received header).
  1. Then check that email's headers once received, especially the
     `Received:` one.
* Also check that the EHLO/HELO SMTP message is not leaking anything
  at the application level:
  1. Start Claws using the panel icon.
  2. Disable SSL/TLS for SMTP in Claws (so take precautions for not
     leaking your password in plaintext by either changing it
     temporarily or using a disposable account).
  3. Run `sudo tcpdump -n -i lo -w dump` to capture the packets before
     Tor encrypts it, then close tcpdump
  4. Check the dump for the HELO/EHLO message and
     verify that it only contains `localhost`: `tcpdump -A -r dump`
  5. Check the `Received:` and `Message-Id` fields in the received
     message: it must not leak the hostname, nor the local IP.
* Make sure Claws Mail use its dedicated `SocksPort` when connecting
  to IMAP / POP3 / SMTP servers by monitoring the output of this
  command:

      sudo watch -n 0.1 'netstat -taupen | grep claws'

# WhisperBack

* I should be able to send a bug report with WhisperBack.
* When we receive this bug report on the tails-bugs mailing-list,
  Schleuder tells us that it was sent encrypted.

# Erase memory on shutdown

- `memlockd` must be running
- After booting from DVD, remove Tails boot medium and check that the
  memory erasure process is started (`Loading new kernel`, at least).
  (automate: [[!tails_ticket 5472]])
- After booting from USB, remove Tails boot medium and check that the
  memory erasure process is started (`Loading new kernel`, at least).

# Root access control

* Check you can login as root with `su` neither with the `amnesia` password nor
  with the `live` one.
* Check that the `$TAILS_USER_PASSWORD` variable, if still existing in the system
  environment after the boot has finished, does not contain the clear text
  password.

# Virtualization support

* Test that Tails starts and the browser launches in VirtualBox.

# I2P

Make sure that I2P is up-to-date, at least if the
[changelogs](https://geti2p.net/en/blog/) mention that
security critical bugs were fixed.

Start I2P by appending `i2p` to the kernel command line.

* Check that I2P starts when a network interface is up:
  - Within 30 seconds you should get the "I2P router console is ready"
    pop-up
  - Start the I2P Browser via "Applications -> Internet -> I2P Browser":
    * You get the "Starting I2P Browser..." pop-up.
    * The router console (<http://127.0.0.1:7657>) opens successfully
      upon success.
    * On exiting I2P Browser, check that its chroot gets properly torn
      down on exit (there should be nothing mounted inside
      `/var/lib/i2p-browser`).
  - After a few minutes you should get the "I2P is ready" pop-up
  - Go to <http://127.0.0.1:7657/i2ptunnelmgr> in the I2P Browser:
    * You should get "Network: Hidden" in the "General" section.
    * The numbers in the "Peers" section of the sidebar should be
      non-zero.
    * Check that you can reach some eepsites within Iceweasel, like
      <http://i2p-projekt.i2p> and <http://forum.i2p>.
  - Check that you can connect to the I2P IRC server through Pidgin
    and the preconfigured IRC account on 127.0.0.1.
* Check I2P failure modes:
  - Router console failure:
    * Boot without network so I2P doesn't start automatically.
    * Block the router console port: `nc -l -p 7657 -t 127.0.0.1`
    * Plug the network
    * You should get the "I2P failed to start" pop-up, and I2P should
      not be running (check with `service i2p status`)
  - Bootstrap failure:
    * Detach the network immediately after getting the "I2P router
      console is ready" pop-up
    * Wait for up to six minutes
    * You should get the "I2P is not ready" pop-up
    * The I2P router console should still be accessible on
      <http://127.0.0.1:7657>

# SSH

* Connecting (by IP) over SSH to a server on the LAN should work. (automate: [[!tails_ticket 9087 desc="#9087"]])

# APT (automate: [[!tails_ticket 8164 desc="#8164"]])

     grep -r deb.tails.boum.org /etc/apt/sources.list*

* Make sure the Tails repository suite in matching the release tag (for example
  the release version number) is in APT sources.
* Make sure the Tails repository unversioned suites (e.g. `testing`,
  `stable` and `devel`) are *not* in APT sources.

<a id="incremental-upgrades"></a>

# Incremental upgrades

* List the versions from which an upgrade paths to this one is described.
  In the `stable` or `testing` branch:

      git grep -l "  version: '\?0.23'\?" wiki/src/upgrade/

* For each description file, open it and verify if it allows incremental upgrade
  or only full upgrade.

* For each previous version from which an upgrade paths is described, install it
  and try to upgrade:
  * For every incremental upgrade path: make sure the resulting updated
    system "works fine" (boots and pretends to be the correct version).
  * For upgrade paths that only propose a full upgrade: make sure the
    user is told to do a manual upgrade.

  If:
  
  * the update-description files have been published on the
    *alpha* channel already (see <https://tails.boum.org/upgrade/v1/Tails/>)
  * and the IUK has been published already (see
    <https://archive.torproject.org/amnesia.boum.org/tails/alpha/>
    and <https://archive.torproject.org/amnesia.boum.org/tails/stable/>):

  then:
  
        echo 'TAILS_CHANNEL="alpha"' | sudo tee --append /etc/os-release && \
        tails-upgrade-frontend-wrapper

  Else, use a local test setup:

  * A web server on the LAN.
  * A copy of `wiki/src/upgrade` from the `stable` or `testing` branch,
    for example in `/var/www/tails/upgrade/v1/Tails/0.14~rc2/i386/stable/updates.yml`
  * A copy of the `iuk` directory of our HTTP mirrors,
    for example in `/var/www/tails/stable/iuk/Tails_i386_0.14-rc2_to_0.14.iuk`.

    To synchronize your local copy:

        torsocks rsync -rt --progress --delete rsync.torproject.org::amnesia-archive/tails/stable/iuk/ /var/www/tails/stable/iuk/

  * Patch `/etc/hosts` in Tails to point to your web server:

        echo "192.168.1.4    dl.amnesia.boum.org" | sudo tee --append /etc/hosts

  * Patch sudo configuration to allow passing arbitrary arguments to
    `tails-upgrade-frontend`:

        sudo sed -i \
            -e 's,/usr/bin/tails-upgrade-frontend ""$,/usr/bin/tails-upgrade-frontend,' \
            /etc/sudoers.d/zzz_upgrade

  * Call the upgrader must be called, from inside the system to upgrade,
    with every needed option to use the local web server rather than the
    online one, for example:

        DISABLE_PROXY=1 SSL_NO_VERIFY=1 \
        tails-upgrade-frontend-wrapper --override-baseurl \
        http://192.168.1.4/tails

# Windows Camouflage

Enable I2P in the boot loader menu, and enable Windows camouflage via
the Tails Greeter checkbox, and then:

* Tails OpenPGP Applet's context menu should look readable
* The Tor Browser, Unsafe Browser and I2P Browser should all use the
  Internet Explorer theme.
* Vidalia should not start.

# Unsafe Web Browser

* Browsing (by IP) a FTP server on the LAN should be possible.

* Google must be the default, pre-selected search plugin.

# Real (non-VM) hardware

`[can't-automate]`

* Boot on bare-metal on USB.
* Boot on bare-metal on DVD.
* Measure boot time (from syslinux menu the GNOME dektop ready - quickly press
  enter in the greeter), then on some reference bare metal hardware, and
  compare with previous version. The new one should not be significantly
  slower to start.

# Documentation

* The "Tails documentation" desktop launcher should open the
  [[getting started]] page (automate: [[!tails_ticket 8788]]):
  - in one language to which the website is translated
  - in one language to which the website is not translated (=> English)
* Browse around in the documentation shipped in the image. Internal
  links should be fine.

# Internationalization

Boot and check basic functionality is working for every supported
language. You *really* have to reboot between each language.

* The chosen keyboard layout must be applied.
* The virtual keyboard must work and be auto-configured to use the same keyboard
  layout as the X session.
* In the Tor Browser:
  - Disconnect.me must be the default, pre-selected search plugin.
  - the Disconnect.me, Startpage and Wikipedia search plugins must be
    localized for the supported locales:

        . /usr/local/lib/tails-shell-library/tor-browser.sh
        supported_tor_browser_locales

## Spellchecking

* Check that every supported language is listed in the list of languages for
  spell checking.
  - Visit <https://translate.google.com/>.
  - Right-click and choose "Check spelling".
  - Right-click and check the list of available languages.
* For a few languages, check the spell checking:
  - Type something in the textarea.
  - Right-click and select a language.
  - Verify that the spelling suggestion are from that language.
* Once [[!tails_ticket 5962]] is fixed, the browser spelling dictionary must be
  localized (for languages that are supported by our branding extension).

# Misc

* Check that Tails Greeter's "more options" screen displays properly
  on a display with 600 px height, preferably in a language that's
  more verbose than English (e.g. French).
* Check that all seems well during init (mostly that all services
  start without errors), and that `/var/log/syslog` seems OK.