summaryrefslogtreecommitdiffstats
path: root/wiki/src/blueprint/Debian_testing.mdwn
blob: 8bbb1e7009687df564f4061b28a11bcb3014e39f (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
[[!meta title="Basing Tails on quarterly snapshots of Debian Testing"]]

Tracking ticket: [[!tails_ticket 12615]]

[[!toc levels=2]]

# The big picture

We've been thinking for a while of being based on quarterly snapshots
of Debian testing. We've used the "porting Tails to Debian Stretch"
cycle to evaluate how it would feel like. See the
[[initial plan & analysis|blueprint/Debian_Stretch/#rolling]].

## Calendar

A two-years Debian release cycle could look like from our perspective,
if we were based on quarterly snapshots of Debian testing:

1. a new Debian stable is released
2. 6 months during which Tails is based on Debian stable that was just
   released, and we can get ready for the next stage
3. 12 months during which Tails is based on quarterly snapshots of
   (non-frozen) Debian testing taken at the beginning of each major
   Tails release cycle
4. 6 months during which Tails is based on a frozen Debian testing
5. go back to (1)

We would be based on a moving target half of the time; the remaining
half of the time we are based on something that doesn't change much.

## Why?

- We'll need to track 2 Debian versions at the same time (and
  continuously forward-port development that was based on the oldest
  one) during 6-7 months maximum, compared to 19 months (Jessie cycle)
  and 10 months (Stretch cycle). We would save lots of time there,
  that could be re-invested in aspects of this proposal that will
  require additional work.
- no need to produce and maintain backports of Debian packages anymore
- support new hardware better
- upstream bugfixes trickle down faster and for free to Tails users
  ⇒ greater incentive for us to fix bugs upstream instead of
  implementing Tails-specific workarounds
- we help stabilizing Debian testing
- the Foundations Team prefers doing the "porting to next Debian" work
  continuously than as a huge disruptive project every two years
- no huge change every 2 years confusing users and creating huge burts
  in the Help Desk and Foundations Team post-release workload
- new features / modern software

This idea is about doing a different kind of work, and doing it in
different places than in the past. It probably won't lower the total
amount of effort we have to do, but it likely will make such efforts
generate less frustration, more happiness, and a warm feeling of being
part of a broader community.

# Fears, concerns & problems we need to address or mitigate

- Security
  * How to keep track of security issues that affect us, and whether
    their fix has been uploaded and has migrated to testing yet?
    See e.g. how security support for Debian testing [used to be
    (briefly) done](http://secure-testing-master.debian.net/), and in
    particular the
    [Vulnerable source packages in the testing suite](https://security-tracker.debian.org/tracker/status/release/testing)
    and the
    [Candidates for DTSAs](https://security-tracker.debian.org/tracker/status/dtsa-candidates).
    At first glance it seems totally doable for Foundations Team
    members to run (e.g. locally) a lightweight fork of Debian's
    security tracker:
     - Problem to solve: list security issues that affect Tails
       (current stable and testing branches) but are fixed in
       current Debian testing and/or sid
     - Point it to our relevant APT snapshots (and perhaps current
       Debian testing or sid instead of Tails devel, to avoid having
       to resolve the "latest" unfrozen snapshot)
     - Add our own annotations about issues we decide to
       ignore/postpone, e.g. using the new
       [`<ignored>` and `<postponed>` tags](https://security-team.debian.org/security_tracker.html#issues-not-warranting-a-security-advisory)
     - Filter packages based on some relevant build manifest so we
       don't ever see packages we don't ship in Tails.
     - Automatically merge from Debian's repo and report to us in case
       of merge conflicts.
  * more freeze exceptions in order to address security issues
  * need to keep our stable branch in an always releasable state
    ⇒ the tracking & cherry-picking of security issues must be done
    continuously, and not only right before a planned freeze or release
  * when the updated package from Debian testing/sid is not
    installable on current Tails stable/testing, we'll have to
    backport security fixes

- Transitions
  * How to deal with
    [[!debwiki OngoingTransitions desc="ongoing transitions"]] that
    block migration of security updates from sid to testing?
  * How to select the right snapshot we shall take, e.g.
    during transitions?
  * Sometimes we'll need to rebuild on testing some packages we want
    to cherry-pick from sid.

- Consequences for users
  * too many software regressions and not well tested new features
  * confused users due to constant incremental changes
  * bigger upgrades on average: according to research done on
    [[!tails_ticket 14622]], we should assume ~600MB large IUKs;
    there will be consequences in terms of:
     - RAM requirements: XXX
     - download time: XXX
     - download reliability ⇒ add a retry/continue mechanism to Tails Upgrader?
  * our users debug Debian testing on top of debugging Tails

- Drawbacks for contributors & the Tails community
  * more frequent regressions in ISO reproducibility
  * harder to extract info from help desk
  * harder for help desk to deal with user requests: having to
    re-learn constantly how things work / what's broken
  * spend too much mental space dealing with always changing software
  * lack of Debian skills internally at the moment ⇒ we'll need to
    invest time learning
  * need to better study the delta between N & N+1 (for help desk,
    release notes, security audits, etc.)

- Communication
  * needs new version numbering scheme
  * feeling of stalled/slower project for users because no big
    all-at-a-time changes
  * how do we deal with our (implicit) hardware support promise? i.e.
    "The hardware you purchase for Tails N.0 will work with N.x"
  * harder to get press write stuff as we'll lack releases with tons
    of changes like 2.0 or 3.0

- Additional Software Packages: will they be pulled from current
  Debian testing or from our snapshot? Can we tell APT to install by
  default from Debian testing, unless it's impossible or problematic
  (e.g. would remove tons of packages) and then fall back to
  installing from our snapshot?

# The plan

**Update**: this was deferred. We'll come back to this in 2018-04 or
2018-05 and either come up with a new plan, or postpone this entirely.

* From now to the end of 2017-11: the Foundations Team tries to port
  the code & test suite during sprints. If the work doesn't fit into
  these sprints then we'll need to draw conclusions.
* At the end of 2017-11:
  1. [[!tails_ticket 14578 desc="Decide"]] whether we want to release
     Tails based on Debian testing
     in 2018-01, 2018-04, or give up and rather release it mid-2019.
     The following assumes "in 2018-01" and will need to be adjusted
     if we decide something else.
  2. The Foundations Team tells technical writers what needs to be
     updated ([[!tails_ticket 14579]])
* November-January: technical writers update the documentation
* January 16: first Tails release based on Debian testing

# Zooming in on specific problems

<a id="doc-update"></a>

## How can the Foundations Team tell Technical Writers what part of our doc needs to be updated?

The first time we'll have to deal with this problem is
[[!tails_ticket 14579]], but it will come back every quarter.

Current implementation:

 - [[!tails_gitweb bin/doc-impacted-by]]
 - [[!tails_gitweb doc-source-relationships.yml]]

We want to automate this as much as we can, but let's be clear:
automation will not fully solve this problem unless we switch to
executable documentation (there is research and tools for that but
we're not there yet). So some amount of manual documentation testing
will be needed. Let's start with strategies that will limit the amount
and frequency of the needed manual documentation testing, and further
below we'll discuss manual testing.

### Changes in our code, test suite, and list of included packages

Whenever we have to adjust our code to keep working on Debian testing,
it's plausible that the corresponding documentation needs an update as
well. Developers could either proactively tell technical writers about
it along the way, or look at `git diff` when requested to sum up what
part of Tails is impacted by the changes.

Similarly, whenever we have to update the test suite, it's plausible
that the corresponding documentation needs an update as well.
Test suite maintainers can then add the updated scenario to a list of
things the technical writers should check (again: either incrementally
or upon request).

And finally, developers or technical writers could look at the diff
between the old and new lists of included packages, to identify what
part of Tails is impacted by the changes.

All this is fine but it requires someone to keep a mental index of
what kind of change impacts what part of our documentation. Not only
this is boring work, but nobody is in a good position to do it, as it
requires a good knowledge of our code base _and_ of our documentation.
So I think we should build and maintain this index incrementally, in
Git instead of in our minds, and use it automatically. Read on :)

### Metadata that encodes the relationship between {source code, test suite, packages} and documentation pages

Whenever we notice that changes in a given file in our source code,
automated test suite scenario, or package, have an effect on a given
documentation page, we could encode this information somewhere, in
a machine-readable format. Then, we could write a program whose input
would be:

 * this metadata
 * 2 Git commitish
 * 2 package lists

This program would output the list of documentation pages that are
affected by the changes.

This set of metadata shall be built and garbage-collected incrementally.

Remaining questions:

 - Where to store this metadata?

   At least the information about included packages lives nowhere in
   our Git tree, so encoding it close to the source is impossible
   (besides, I'd rather not have to extract metadata from comments in
   source code written in N different programming languages). So we
   have main options:

   1. Store it in the affected documentation pages themselves,
      (HTML comments? ikiwiki directives? any other idea?)
      Advantages:
      - No work required when renaming pages.
      - Easier to maintain by tech writers as they can see this info
        while working on the doc, which increases the chances they keep
        it up-to-date.
      - Lightweight: no need to explicitly write the doc page name.

   2. Store it externally, in a dedicated file. Advantages:
      - Probably nicer format to read and edit, both for humans and
        computers, than whatever we would embed in doc pages.
      - Does not affect translators: anything we would put in doc pages
        will be extracted to PO files; so storing this metadata there
        would waste translators' time (either because they uselessly
        translate it, or because they paste the English string to reach
        100% translation status and keep their dashboard of pages
        needing attention usable).
      - Works for encoding which sources files impact screenshots.

   intrigeri & anonym discussed that and were torn. The advantages of
   the first option are very compelling, but the drawback it has wrt.
   translators seems to be a deal-breaker. So the second option
   appears to be the best one.

   XXX: sajolida?

 - Defining sets of sources?

   Tails user-facing features are built from a not-that-small number
   of source files. It could be useful to express the fact that
   changes to any of these source files (e.g. the ones that define the
   "Tor Browser" feature) may affect a given doc page, without having
   to list them all for every affected page.

 - Defining sets of doc pages?

   Similarly, it might be that some doc sections share most of their
   dependencies. It could be useful to define named groups of pages.

 - Who shall we optimize the data format for?

   In practice, developers are the ones who know where a given
   user-visible change comes from, so whenever a needed doc update was
   missed, most likely the tech writers will ask the Foundations Team
   either to add the missing metadata, or what exactly caused that
   user-visible change so they can add it themselves.

   I (intrigeri) am really not sure which of these two situations will
   be the most common one, so I'm wary of optimizing too much for one
   of these two groups at this point.

 - Mapping source → doc, or doc → source?

   Update: a format where we define a list of dependencies between
   X doc pages and Y source files, without explicitly mapping one to
   the other, would avoid having to choose too early, which would be
   nice. I'll simply do that.

   Internally, the code that uses this metadata will care about
   "source → doc", but there's no reason why the data format itself
   should be optimized for this program. Let's instead optimize it for
   the humans who will maintain the metadata :)

   "doc → source" expresses a "depends on" / "is affected by"
   relationship, which is well suited when fixing the metadata after
   a postmortem analysis of "why was that doc page not updated before
   the release?". OTOH "source → doc" expresses an "impacts"
   relationship, which is well suited to proactive adjustments of the
   metadata, e.g. when a developer realizes their proposed branch will
   impact the doc. I (intrigeri) am pretty sure that the first
   situation will be the most common one for a while until we get used
   to this metadata and the second one slowly takes over.


Example (YAML):

	# doc → source oriented
	- page: doc/first_steps/accessibility
	  files:
	    - config/chroot_local-includes/etc/dconf/db/local.d/00_Tails_defaults
	  packages:
	    - caribou
	    - dasher
	    - orca
	- page: doc/first_steps/startup_options
	  package: tails-greeter
	  files:
	    - config/binary_local-hooks/*grub*
	    - config/binary_local-hooks/*syslinux*
	    - config/chroot_local-includes/etc/dconf/db/local.d/00_Tails_defaults
	- page: doc/first_steps/startup_options/administration_password
	  package: tails-greeter
	- page: doc/first_steps/startup_options/bridge_mode
	  packages:
	    - obfs4proxy
	    - tor
	  file: config/chroot_local-includes/usr/share/tails/tbb-sha256sums.txt
	  tests: tor_bridges.feature

	# "source → doc" oriented
	- package: gnome-disk-utility
	  pages:
	    - doc/first_steps/persistence/change_passphrase
	    - doc/first_steps/persistence/check_file_system
	- package: tails-persistence-setup
	  tests: persistence.feature
	  pages:
	    - doc/first_steps/persistence/configure
	    - doc/first_steps/persistence/delete
	- package: openpgp-applet
	  tests: encryption.feature
	- package: tails-greeter
      files:
	    - features/images/TailsGreeter*
	  pages:
	    - doc/first_steps/persistence/use
	    - doc/first_steps/startup_options
	    - doc/first_steps/startup_options/*


### Automated testing

As said above, _changes in_ our automated test suite can already help
identify the bits of documentation that need an update. To boost this
capability we could:

 - Add steps to our existing tests to validate the screenshots we have
   in our documentation. This seems doable but not trivial:
    - Screenshots embed context, e.g. window placement and elements
      that change all the time; so we would have to extract the
      relevant part of each screenshot (for example based on
      coordinates encoded somewhere).
    - Credentials used when taking screenshots (e.g. Pidgin) must be
      the ones used by the isotesters.
    - Screenshots in our documentation are resized to 66% so we need
      to resize them back. This is quite easy, but the information
      lost in the double-resizing process may prevent Sikuli from
      finding it on the screen.
 - Add automated tests that specifically check things we document.
 - Try to make our Gherkin scenarios closer to end-user documentation.

### Manual testing of the documentation

Which part of the documentation shall be tested, when, and by whom?

The sources of information listed above should help focus this effort
on the relevant parts of our documentation. 

In many cases, a quick look will be enough, and following the entire
doc page to the letter won't be needed. But likely stricter testing
will be needed from time to time.

Random ideas:

 - Aim for automation: feed the aforementioned metadata store with
   information gathered during manual testing, so that we need less
   and less manual testing as time goes.

 - Prioritize: focus on core pages.

 - Spread the load between different release, i.e. test some pages for
   some release and some others for another release.

### Dismissed options

#### Manual testing of Tails releases

In theory our manual test suite could also help: some of the manual
tests could be performed by following our end-user documentation to
the letter. But I've not been able to find any single candidate test
case for this, so let's forget it.

#### Users and Help Desk

Users and Help Desk are also in a good position to notice outdated
bits and report them back, but they can only do so after the fact,
while ideally we'd rather update the documentation _before_ the
corresponding changes are released to our users. So if we receive such
notifications, e.g. at RC time, fine, but let's not count on it
too much.