summaryrefslogtreecommitdiffstats
path: root/wiki/src/blueprint/automated_builds_and_tests.mdwn
blob: 8c76ae04f67ec76572b50d979b2ecd1639a535b8 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
[[!meta title="Automated builds and tests"]]

[[!toc levels=2]]

Rationale
=========

An automated build and test environment would be pretty useful to
ensure a few facts:

- new code does not break anything
- new build tools (`live-build`) and included software
  (`live-config`, `live-boot`) don't break anything
- updates pushed to the APT repositories we don't control (e.g. Debian's)
  don't break anything
- the quality of our releases is good enough
- power users get to test early images built from feature branches,
  or from `experimental`
- we can ask bug reporters to test an image built from a bugfix
  branch, and see if it fixes their problem

What we have
============

* A Jenkins instance running on lizard, that builds a few branches
  whenever they are pushed. Results [are available
  publicly](http://nightly.tails.boum.org/).
* A partially automated test suite:
  - [[documentation|contribute/release_process/test/automated_tests]]
  - [[setup|contribute/release_process/test/setup]]
  - [[usage|contribute/release_process/test]]
  - [[test cases|contribute/release_process/test]] that are not
    implemented yet.

Roadmap
=======

0. Complete phase one (**build and publish ISOs automatically**):
   fix all child tickets of
   [[!tails_ticket 5324]].
0. Run the test suite automatically on autobuilt ISOs:
   [[!tails_ticket 5288]]
0. See other tickets in the *Continuous Integration* category,
   and organize the next steps.

What we need
============

Automated tests
---------------

Ideally, every automatically built ISO should be automatically tested,
and the results displayed online.

Infrastructure
--------------

Putting automatic builds and automatic tests on the same physical
machine (while using
virtualization to separate concerns) saves hosting and bandwidth costs:
fresh images can be tested without being transferred.

In order to be able to build Tails in memory (which greatly speeds up builds)
and to properly test HIGHMEM memory wipes, the test server should have at
least 32 GB. Using KVM inside a KVM
virtual machine is supported by the "nested KVM" feature.
VirtualBox (Vagrant) in KVM is not supported as of March 2013.

We could set a threshold of tests that have to be successful before publishing
one daily build image. This would prevent too-broken images to spread out and
potentially harm users.

The setup of the automated build and test server should be done
with public Puppet modules.

More specific pages
===================

[[!map pages="blueprint/automated_builds_and_tests/*"]]