summaryrefslogtreecommitdiffstats
path: root/wiki/src/contribute/release_process/test.mdwn
blob: 2048e4da991cf7b98bc703cd07fa9bcbf9a360a2 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
[[!toc levels=1]]

Some [[test results]] that might be useful to keep are saved.

# Changes

Keeping an eye on the changes between released versions is one of the
many safeguards against releasing crap.

## Source


Compare the to-be-released source code with previous version's one e.g.:

Boot the candidate ISO and find the commit it was build from with the
`tails-version` command.

Then, from the source tree, see the diff:

	git diff <old tag>..<ISO commit>

e.g. `git diff 0.17..06fa1ab80d55c9f29274b7459bd198edb1a8d53d`

## Result

Compare the list of bundled packages and
versions with the one shipped last time e.g.:

	/usr/bin/diff -u \
	    wiki/src/torrents/files/tails-i386-0.16.packages \
	    tails-i386-0.17.packages \
	    | wdiff --diff-input  --terminal

Check the output for:

- new packages that may cause harm or make the images unnecessarily
  big
- packages that could be erroneously removed
- new versions of software we might not have audited yet (including:
  does the combination of our configuration with software X version
  Y+1 achieve the same wished results as with software X version Y?)

## Image size

Check the image size has not changed much since the last release.

# Automated test suite

Our long term goal is to eliminate the manual test suite (except the
parts which require real hardware) and have the automated test suite
run all our tests. It's design, and how to write new tests, are
documented on a [[dedicated page|test/automated_tests]].

## Running the automated test suite

See [[test/setup]] and [[test/usage]].

## Automated test suite migration progress

The manual test suite below either contains tests that cannot be
automated, has no automated test implemented yet, or has a test
implemented, but it either hasn't been reviewed, had a confirmed pass
by someone other than the test author, or has issues. The latter is
tracked by tickets prefixed with `todo/test_suite:`.

# Iceweasel

* does the exposed User-Agent match the TBB's one?
  (connect to a website you can check the access logs for)
* Running `getTorbuttonUserAgent` should produce the useragent set by the
  Torbutton version installed and used in Iceweasel.
* Browsing (by IP) a HTTP or HTTPS server on the LAN should be possible.
* Browsing (by IP) a FTP server on the LAN should be possible.
* Does playing HTML5 videos work? In particular, (due to its popularity)
  do [youtube](http://www.youtube.com) videos work (once
  [[!tails_todo easier YouTube]] is fixed -- currently object unblocking in
  NoScript is required)?
* Compare the fingerprint of Tails and the latest TBB using at least
  <https://panopticlick.eff.org/> and <http://ip-check.info/>, and
  update the [[fingerprint section|support/known_issues#fingerprint]] of
  the known issues page.

# Pidgin

* pidgin: is an IRC session really torified?
 - if you are running an IRC server: check there
 - else: see if the connection to the IRC server appears in Vidalia
   connections list
* Check OTR is working.
* Check that IRC is working with the default OFTC profile.
* Check that XMPP is working with a new test profile.
* Check if pidgin doesn't leak too many informations on replying to different
  CTCP requests:
  * Start Tails and launch pidgin once done. Connect to an IRC server where you're already on
  * try to `/ctcp <Tails_account_nick> COMMAND`
    with a client that supports sending CTCP commands properly --
    Pidgin does not, apparently
    (for a list of commands, see [this page](http://www.wikkedwire.com/irccommands)).
    See the [[!tails_ticket 5823 desc="current known-broken state"]].

# Tor

* The version of Tor should be the latest stable one.
* firewall: is the Tor-enforcement effective?
 - check output of `iptables -L -n -v`
 - check output of `iptables -t nat -L -n -v`
 - try connecting to the Internet after unsetting `$http_proxy` and
   `$HTTP_PROXY` using a piece of software that does not obey the
   GNOME proxy settings, *and* is not explicitly torified in Tails:
   `unset http_proxy ; unset HTTP_PROXY ; wget --no-proxy
   http://monip.org` (should result in a "`Connection refused`".
* firewall: is IPv6 traffic blocked?
 - check output of `ip6tables -L -n`
 - at a place with working IPv6: try connecting to a known-working
   IPv6-enabled server on its IPv6 address over TCP and icmp6.
* is `/etc/resolv.conf` OK both before/after DHCP has been set up? it should
  *always* read `nameserver 127.0.0.1`
* [[doc/first_steps/startup_options/bridge_mode]] should work:
  1. Boot with the `bridge` option.
  1. Set up an administrator password.
  1. Enter a bridge or a few.
  1. Use the Internet.
  1. Check from time to time with `netstat` that the only outgoing
     direct network connections go to the configured bridge(s).
* Verify that all destinations reached from an intensive Tails session
  are tor routers or
  authorities:
  1. Boot Tails without the network in.
  1. Start dumping your whole session's network activity with `sudo
     tcpdump -n -i any -w dump` (or better, do the dump on another machine,
     or on the host OS if Tails is running in a VM).
  1. Plug the network.
  1. Wait for Tor to be functional.
  1. Save the `cached-microdesc-consensus` out of the VM (it's needed
     to analyze the network dump later on).
  1. Do *a lot* of network stuff (why not run do this while doing all
     the other tests **but** I2P and the unsafe browser, which would
     show many false positives?)
  1. Then check all destinations, e.g. by using tshark and the script below:

     	# set DUMP to the output of tcpdump above
     	DUMP=dump
     	# set CONSENSUS to Tor's consensus from the Tails session
     	CONSENSUS=cached-microdesc-consensus
     	NODES=$(mktemp)
     	awk '/^r / { print $6 }' ${CONSENSUS} > ${NODES}
     	# Note that these default directory authorities may change! To be
     	# sure, check in Tor's source, src/or/config.c:~900
     	DIR_AUTHS="
     	128.31.0.39
     	86.59.21.38
     	194.109.206.212
     	82.94.251.203
     	76.73.17.194
     	212.112.245.170
     	193.23.244.244
     	208.83.223.34
     	171.25.193.9
     	154.35.32.5
     	"
     	tshark -r ${DUMP} -T fields -e ip.dst | sort | uniq | \
     	while read x; do
     	    ip_expr=$(echo ${x} | sed -e "s@\.@\\\.@g")
     	    if echo ${DIR_AUTHS} | grep -qe "${ip_expr}"; then
     	        continue
     	    fi
     	    if ! grep -qe "^${ip_expr}$" ${NODES}; then
     	        echo "${x} is bad"
     	    fi
     	done
     	rm ${NODES}

     Note that this script will produce some false positives, like your
     gateway, broadcasts, etc.

## Stream isolation

See our [[stream isolation design
page|contribute/design/stream_isolation]] for details such as port
numbers, that are not duplicated here to avoid desynchronization.

Assumptions for the following tests: first, Tor stream isolation
features properly do their work; second, our `torrc` sets the right
`SocksPort` options to implement what we want.

* Make sure Claws Mail use its dedicated `SocksPort` when connecting
  to IMAP / POP3 / SMTP servers.
* Make sure these use the `SocksPort` dedicated for Tails-specific applications:
  - htpdate
  - tails-security-check
  - incremental updates
* Make sure iceweasel uses its dedicated `SocksPort`.
* Make sure other applications (Polipo, Gobby 0.4 and 0.5, SSH, whois)
  use the default system-wide `SocksPort`.
* Make sure a random application run using `torify` and `torsocks`
  (e.g. `/usr/bin/gobby-0.5`) uses the default system-wide `SocksPort`.

# Use of untrusted partitions

* is any local hard-disk swap partition used as swap?
  boot on a (possibly virtual) machine that has a cleartext swap
  partition not managed by LVM. This swap partition must not be used
  by Tails.
* is a persistence volume on a local hard-disk partition used?
  (Hint: setup a libvirt USB disk with GPT and a partition labeled
  `TailsData`, set the `removable` flag on it, check that
  tails-greeter proposes to enable persistence. Then remove the
  `removable` flag, and check that tails-greeter does not propose to
  enable persistence anymore.)

# Claws

* Check mail over IMAP using:
  - a "clearnet" IMAP server.
  - a hidden service IMAP server (e.g. TorMail, jhiwjjlqpyawmpjx.onion, or
    RiseUp, xyvp43vrggckj427.onion).
* Send an email using:
  - a "clearnet" SMTP server.
  - a hidden service SMTP server (see above).
* Check that the profile works and is torified (specifically the
  EHLO/HELO SMTP messages it sends):
  1. Send an email using Claws and a non-anonymizing SMTP relay.
  1. Then check that email's headers once received, especially the
     `Received:` and `Message-ID:` ones.
* Also check that the EHLO/HELO SMTP message is not leaking anything
  with a packet sniffer:
  1. start Claws using the panel icon.
  1. Disable SSL/TLS for SMTP in Claws (so take precautions for not
     leaking your password in plaintext by either changing it
     temporarily or using a disposable account).
  2. Run `sudo tcpdump -n -i lo -w dump` to capture the packets before
     Tor encrypts it, then close tcpdump, and check the dump for the HELO/EHLO message and
     verify that it only contains `localhost`.

# Whisperback

* can a bug report e-mail be sent?
* is it correctly encrypted?

# Time

1. Boot Tails without a network cable connected.
   (e.g. `virsh domif-setlink tails-dev 52:54:00:05:17:62 down`.)
2. Set an administration password.
3. set the time to an obviously wrong one:

           date --set="Mon, 01 Mar 2000 15:45:34 - 0800"

4. Connect the network cable.
   (e.g. `virsh domif-setlink tails-dev 52:54:00:05:17:62 up`)

=> the date should be corrected and Tor/Vidalia should start
correctly.

# erase memory on shutdown

- check that `memlockd` is running
- check that `udev-watchdog` monitors the right device
  ([[!tails_todo test_suite:_udev-watchdog_device]])
  * on USB
  * on DVD
- After booting from DVD, remove Tails boot medium and check that the
  memory erasure process is started (`Loading new kernel`, at least).
  The USB tests are bellow, in the USB installer section.

Testing that the needed files are really mapped in memory, and the
erasing process actually works, involves a more complicated
process that is worth [[a dedicated page|test/erase_memory_on_shutdown]].
`[needs-fix: erase_memory]` ([[!tails_todo test_suite:_anti-tests]])

# Root access control

* Check you can login as root neither with the `amnesia` password nor
  with the `live` one.
* Boot Tails without setting a password and check you can't get
  PolicyKit administrative privileges (e.g. run a command with pkexec),
  neither with the `amnesia` password nor
  with the `live` one.
* Boot Tails with a password and check you can get PolicyKit
  administrative privileges.
* Check that the $TAILS_USER_PASSWORD variable, if still existing in the system
  environment after the boot has finished, does not contain the clear text
  password.

# Virtualization support

* test in VirtualBox

# I2P

* Make sure that I2P is up-to-date, at least if the
  [changelogs](http://www.i2p2.de/announcements.html) mention that
  security critical bugs were fixed.
* Check that "Applications -> Internet -> I2P" works:
  - You get the "Starting I2P..." pop-up.
  - The router console opens in Iceweasel upon success.
  - You get the "I2P failed to start" pop-up on failure (e.g. no
    network so tordate failed).
* Check that I2P connects to the network:
  - The numbers in the "Peers" section of the router console should be
    non-zero.
  - You should get "Network: Hidden" in the "General" section.
* Check that you can reach some eepsites within Iceweasel, like
  <http://www.i2p2.i2p> and <http://forum.i2p>.
* Check that you can connect to the I2P IRC server through Pidgin and
  the preconfigured IRC account on 127.0.0.1.

# Git

* clone a repository over `git://`
* clone a repository over `https://`
* clone a repository over SSH

# SSH

* Connecting over SSH to a server on the Internet should work (and
  appear in Vidalia's connections list).
* Connecting (by IP) over SSH to a server on the LAN should work.
* Connecting to a sftp server on the Internet using GNOME's "Connect
  to a server" should work.

# APT

* Make sure the Tails repository suite matching the release tag is in
  APT sources.
* Make sure the Tails repository unversioned suites (e.g. `testing`,
  `stable` and `devel`) are *not* in APT sources.

# USB Installer/Upgrader

All this was successfully reviewed at commit cdbbf4d8.

After each of the following tests:

* The installed or upgraded Tails medium shall be successfully booted.
* The installed or upgraded Tails medium shall have the "emergency
  shutdown on boot medium removal" feature working: remove Tails boot
  medium and check that the memory erasure process is started
  (`Loading new kernel`, at least).
* The TailsData partition must survive
  upgrades.

Tests to run:

* Test "Clone & Install":
  - from USB.
  - onto a USB stick that has a MBR partition table, and no
    partition at all (regression test)
  - onto a USB stick that already has an old-fashioned hybrid cat'd
    Tails on it
* Test "Clone & Upgrade" (onto a USB install containing an older
  Tails):
  - from DVD.
  - from USB.
  - onto a USB stick that already has a old-fashioned cat'd hybrid
    Tails on it: should warn this action is not supported, and direct
    the user to the "Clone & Install" operation mode.

# Incremental updates

The following should be tested once incremental updates are ready for
real-world deployment.

Try every update path supported by the generated update-description
files:

* for every incremental update paths: make sure the resulting updated
  system "works fine" (what does that mean? do we want to run the full
  test suite on these? **FIXME**)
* for updates that only propose a non-incremental paths: make sure the
  user is guided just fine (that is? **FIXME**)

Given these are not published yet, a local test setup is needed:

* a web server
* a replacement for the Tails website's `/update/` tree with
  update-description files in there, e.g.
  `/var/www/tails/update/v1/Tails/0.14~rc2/i386/stable/updates.yml`
* a replacement for the Tails HTTP mirrors' `iuk` directories,
  e.g. `/var/www/tails/stable/iuk/Tails_i386_0.14-rc2_to_0.14.iuk`

Also, the updater must be called, from inside the system to update,
with every needed option to use that rather than the online published
stuff: **FIXME**.

# Persistence

* Activate persistence on a Tails USB install with all presets on
  (also done by the automated test suite,
  but required to test what follows until it's green too).
* Try read-write mode. Make sure that changes do survive
  reboot.
* Try read-only mode. Make sure that persistent files are writeable,
  but that no changes survive
  reboot.
* Turn off some persistence presets, reboot, and make sure they are
  not activated.
* Delete a persistent volume, reboot, the persistence option should
  not appear in the Greeter anymore.

# Windows Camouflage

Enable Windows XP camouflage via the Tails Greeter checkbox and:

* gpgApplet's context menu should look readable
* iceweasel should use a Internet Explorer theme

# Unsafe Web Browser

* On start, if no DNS server was configured in NetworkManager
  (e.g. if there's no network connection), there must be an error.
* Once started, check that:
  - it has no scary red theme when Windows Camouflage is activated.
  - the iceweasel instance runs as the `clearnet` user.
  - it has no proxy configured.
  - no extensions are installed.
  - there are no bookmarks.
* On exit, check that:
  - make sure that its chroot gets properly teared down on exit (there
    should be nothing mounted inside `/var/lib/unsafe-browser`).

# Real (non-VM) hardware

`[can't-automate]`

* Boot on bare-metal on USB.
* Boot on bare-metal on DVD.
* Measure boot time (from syslinux menu the GNOME dektop ready - quickly press
  enter in the greeter), then on some reference bare metal hardware, and
  compare with previous version. The new one should not be significantly
  slower to start.

# Documentation

* Check that links to the online website (`Mirror:`) at the bottom of
  bundled static web pages are working. Else, it probably means the
  wiki was not built with a recent enough ikiwiki.
* Browse around in the documentation shipped in the image. Internal
  links should be fine.

# Internationalization

Boot and check basic functionality is working for every supported
language.

* The chosen keyboard layout must be applied.
* The virtual keyboard must work and be auto-configured to
  use the same keyboard layout as the X session.
* Once [[!tails_todo fix_localized_iceweasel_search_engine]] is fixed, the
  iceweasel search engine must be localized (for languages we ship
  a localized searchplugin for).
* Once [[!tails_todo select_localized_spelling_dictionary_in_Iceweasel]] is
  fixed, the iceweasel spelling dictionary must be localized (for
  languages that are supported by our branding extension).

# Misc

* Check that all seems well during init (mostly that all services
  start without errors), and that dmesg seems ok.
* MAT should be able to clean a PDF file.
* Browsing (by hostname) a FTP server on the Internet should be
  possible using GNOME's *Connect to server* feature
  (once [[!tails_todo fix Internet FTP support]] is fixed)
* The Tails signing key in `/usr/share/keyrings/tails-keyring.gpg`
  should be up-to-date (that is, neither it nor one its subkeys must
  have expired, or be about to expire any time soon).