summaryrefslogtreecommitdiffstats
path: root/wiki/src/blueprint/Endless_upgrades.mdwn
blob: 87ececd4957bb6f354b48dcb6fcab0cb8223fa05 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
[[!meta title="Endless upgrades"]]

Currently automatic upgrades have the limitation that only some N
number of them can be performed in a row before the system partition
runs out of space and automatic upgrades cannot be done further, but a
full, manual upgrade is needed.

Automatic upgrades should be possible forever. This would:

  - Improve UX as currently full upgrades are one of the most complicated and
    long scenario, requiring at least 2 USB sticks and 2 restarts.

  - Improve security by eliminating the need for recurring verifications
    of ISO images and reducing cross-contamination by limiting the
    number of times you might want to clone or rely on another operating
    system for downloading or upgrading.

[[!toc levels=2]]

# Different approaches to endless upgrades

While examining these approaches, let's forget about IUKs for a while,
and only consider full automatic upgrades ([[!tails_ticket 7499]]):

## Extract upgrade from ISO

Keeping our current partition layout and filesystems, a full upgrade of
a running system can be done by extracting the required parts from the
new version's .iso. Here we'd need at least 3 times the size of an
installation available:

1. one for the old, running installation
2. one for the new ISO
3. one for the extracted contents of the new ISO

We would unlink the old squashfs, kernel, initrd etc. then copying in
their place new version's ones, so the new version will run next time
we reboot.

### Disk images instead of ISOs

Note that if we switch to using disk images as the primary way to
install Tails ([[!tails_ticket 8550]], comment 9) we can similarly
extract the needed parts just like above: replace "(ISO|.iso)" with
"disk image" and it still works.

## Streaming decompression

An improved version of the above idea would be to serve "upgrade
packs" using some compression method friendly to streaming
decompression, so the need to download a the full archive (ISO or disk
image above) is avoided, which reduces the amount of temporary disk
space needed from 3 to 2 times the size of an installation when
performing the upgrade.

## ISOBoot

Grub and syslinux supports booting directly from an ISO so we could
switch to a strategy where Tails USB/SD installation means that we
dump the ISO image into the system partition, and configure the
bootloader to boot from it. If we make the system partition large
enough to store two ISOs (= 2 times the size of an installation, when
comparing o the above methods) we can keep the older, running version
in place while downloading the new one, and then safely switch to the
new one at next boot. Since the old version is kept, there's a
fallback in case the new version doesn't boot. Older versions can be
cleaned up at the next upgrade.

Depending on how Grub/syslinux does the ISOBoot, this may make all
running Tails instances more consistent, i.e. we now do not have the
three cases of what type of medium Tails has booted from: DVD; USB
drive; or SD card -- in all cases we have a ISO9660 filesystem. OTOH,
the magic employed for the ISOBoot may hide too much about underlying
real device, which we still probably will need information about.

Some more details (some perhaps wrong) can be read in comment 16 on
[[!tails_ticket 7496]].

## Apply upgrade in initramfs

By storing the ISO (or disk image) that the upgrade is gonna be
extracted from on a scratch partition, and then apply the upgrade in
initramfs on next boot, we only need 2 times the size of an
installation in total, and can safely download the ISO to the scratch
partition while keeping the system partition read-only.

Ideally, let's also save the files used to authenticate the upgrade on
the scratch partition, and make check the authenticity during
initramfs, before starting the upgrade.

Also, with the scratch partition we will never have to mount it
read-write outside of initramfs, it can be made read-only in a
slightly stronger sense on the block dev level. This is not really a
security feature, but more a foolproof things, but why not?

<a id="non-nih"></a>

## How others do i.e. non-NIH solutions

This survey will be updated in a while with [[!tails_ticket 15277]].

* yocto project's
  [comparison of system update mechanisms](https://wiki.yoctoproject.org/wiki/System_Update)
* [RAUC](https://rauc.readthedocs.io/en/latest/)
  - manages
    [boot loader](https://rauc.readthedocs.io/en/latest/integration.html#interfacing-with-the-bootloader)
    and A/B partition setups, e.g.
    [GRUB](https://rauc.readthedocs.io/en/latest/integration.html#grub)
    ([example config file](https://github.com/rauc/rauc/blob/master/contrib/grub.conf))
    and
    [EFI](https://rauc.readthedocs.io/en/latest/integration.html#efi)
  - systemd integration and D-Bus API
  - super flexible
  - no support for binary delta updates yet, but there's
    a [RFC for casync support](https://github.com/rauc/rauc/pull/217)
    which downloads only missing bits and downloads/applies with
    streaming (no intermediate download of images)
  - ITP: [[!debbug 916210]]
* [Qt's Over-The-Air Update](https://doc.qt.io/QtOTA/) aka. QtOTA,
  that uses OSTree
* XXX: other OSTree based designs/implementations?
* XXX: other [casync](http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html) ([website](https://github.com/systemd/casync))
  based designs/implementations?
* XXX: how efficient is casync with SquashFS? in other words, do we
  need to unpack the rootfs on the target system partition?
* Chrome OS keeps the running system partition read-only, has _two_
  system partitions, and the update is applied to the non-running one.
  This has nice additional properties like allowing one to rollback if
  the upgraded system doesn't start properly. More information about
  their design and implementation:
  - [disk format](https://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format)
  - [file system and autoupdate system](https://www.chromium.org/chromium-os/chromiumos-design-docs/filesystem-autoupdate)
  - [Android's A/B System Updates](https://source.android.com/devices/tech/ota/ab_updates.html)
* [Endless ostree builder](https://github.com/cosimoc/deb-ostree-builder)
* [[!debbug 824649 desc=" ostree: document how to prepare and update a .deb-based system root"]]

### Not suitable

(but could be good sources of inspiration!)

* EndlessOS'
  [OSTree based upgrader](https://github.com/endlessm/eos-updater)
  - supports separate poll/fetch/apply operations (so presumably we
    could make the relevant partition read-write only after fetching,
    as long a we can fetch to a tmpfs)
  - daemon + D-Bus API
  - systemd integration
  - supports Flatpak
  - likely too many EndlessOS-specific assumptions… and written in
    C so difficult to adapt to our needs
* [mender](https://mender.io/): downloads full update
* [swabic's swupdate](https://sbabic.github.io/swupdate/): downloads
  full update; binary delta updates are
  [on the roadmap](https://sbabic.github.io/swupdate/roadmap.html#binary-delta-updates)

# Auxiliary ideas

## Read-write system partition while the network is enabled

Currently we keep the system partition read-only as long as the
network is enabled. E.g. we download IUKs into the tmpfs, and when
it's finished we disable the network and only then remount the system
partition read-write to apply the upgrade.

This is problematic for full upgrades, since we cannot assume that
users' systems has over 1 GiB of free RAM available but need to
download it to disk, but cannot use the system partition. In the
"extract upgrade from ISO/Disk image" case we can simply introduce a
scratch partition where the temporary download is stored without loss
of disk usage efficiency (we still need 3 times an installation). But
for the ISOBoot and streaming decompression cases we'd then go from 2
times to 3 times the size of an installation.

It might be worth having a discussion about whether preventing the
network to be used while the system partition is mounted read-write
actually brings us any security. After all, if the system is
compromised while downloading the upgrade, it will still be
compromised when the download is finished and the system partition is
remounted read-write, and it can infect that part then.

Note that with A/B system updates this discussion is moot: the
bootloader configuration update is the only part of the process that
requires read-write access.

## What about IUKs?

If we want to support IUKs at the same time as full upgrades, we need
to allocate even more space to the system partition. Also, we need to
carefully consider the UX part of this.

For instance, we could always give users the choice to do a full or
incremental upgrade unless installing the incremental upgrade will
result in too little disk space remaining for a full upgrade to be
done *next* time -- in this case we only allow full upgrades, to
prevent users from locking themselves in a situation where they cannot
upgrade further. When both options are available, we could give users
the suggestion that they should do a full upgrade if they currently
have a lot of time and/or a fast connection.

## Self-contained verification

An interesting option for Tails installation verification
([[!tails_ticket 7496]]) is to contain all the bits needed to verify
an installation's integrity. For instance, in the ISOBoot case we'd
include the .sig for the ISO on as well as the .iso, so an external
verifier simply can verify the .sig instead of needing an external
known-good .iso to compare with.

The ISOBoot case is the simplest, probably, as the other methods would
require signed manifests of files with the expected hashes (or
similar), which of course is not such a big deal to provide.

The ISOBoot case would still require a manifest of files we extract
from the ISO (bootloader code and configuration), but indeed that
manifest could be in the ISO itself, and thus its authenticity and
integrity would be signed/verified at the same time as the ISO.

<a id="iuks-on-steroids"</a>

<a id="single-squashfs-diff"></a>

# Temporary stop gap: stack one single SquashFS diff when upgrading

(This idea was originally conceived in the comments of [[!tails_ticket
11131]].)

Without any major redesign of our current incremental upgrade system
(i.e. comparatively cheap) we can allow users upgrade only through
IUKs for much longer, probably throughout a whole Tails series,
e.g. all of Tails 2.x, which we'll use for our examples below since we
moved to 3.x already so we have all data we need to test this idea.

## Basic idea

### Inside Tails

* Tails Upgrader fetches the UDF for the version of Tails that was
  originally installed (instead of the version of the running Tails),
* We remove any previous IUK (and reference to it in `Tails.module`)
  when a new IUK is installed.

### Release management

When we release version M.N, then, for all previous versions M.X since
M.0, we generate a M.X_to_M.N IUK.

(Actually, it is not necessary to choose the first release of a series
(like M.0) as the "synchronization point". We could add another one
half way through the series, or make every tenth release a
synchronization point, or we could do it when when the IUK simply
grows too big. We have lots of options to balance this according to
our needs, but the first release of a series seems like a technically
reasonable choice according to the data we have for the 2.x series
(see below), but also nice from the user's PoV if the only time a
manual upgrade is needed is when upgrading between series.)

### Example

1. I install Tails 2.2 on a USB stick.
2. Later when Tails 2.3 is released I upgrade to it via the 2.2_to_2.3
   IUK (i.e. identical to our original scheme).
3. When Tails 2.4 is released I upgrade to it via the 2.2_to_2.4 IUK
   (whereas in our original scheme we would stack 2.3_to_2.4 on top of
   2.2_to_2.3) and remove the 2.2_to_2.3 IUK (whereas in our original scheme
   we never remove IUKs).

Notice how step 3 can be repeated for each new release, so that we in
the end reach the final release of the series, Tails 2.12. And notice
how we in step 1 could have installed any 2.x release.

## Problems due to IUK size

We'll be dealing with larger IUKs (because generally more and more
changes accumulate between different Tails versions) and only have one
present at a time, with the exception being while we are installing a
new IUK (then two will be present). Let's look at the IUK sizes
towards the end of the 2.x series as if the last manual install was
for Tails 2.0, which is the excepted worst case:

    569M Tails_i386_2.0_to_2.9.iuk
    678M Tails_i386_2.0_to_2.10.iuk
    682M Tails_i386_2.0_to_2.11.iuk
    634M Tails_i386_2.0_to_2.12.iuk

The IUK size is involved in at least four concerns:

* Disk space needs on mirrors: for 2.x there were 14 non-rc/beta/alpha
  Tails releases, so there would be 13 IUKs for the last release
  (2.12). So using the worst-case as the average case we get the upper
  bound of the space needs on each mirror: 14*682 MB ~= 10 GB.
* Disk space needs on the Tails system partition: we need to be able
  to store two large IUKs. Apparently the worst case for 2.x is the
  2.10 to 2.11 upgrade if 2.0 was the version originally installed:
  - The 2.0 installation itself uses 1082 MB, so upgraded to 2.10 the
    total usage is 1082 MB + 678 MB = 1760 MB.
  - Due to the Upgrader requiring 3x the size of the IUK on the Tails
    system partition, the actual requirement is 1760 MB + 3*682 MB =
    3806 MB of free disk space. That wouldn't have worked during the
    2.x times, when the system partition was 2.5 GB, and it barely
    would work with our current 4.0 GB partition. OTOH, during the 2.x
    series we shipped two kernels, while we don't in 3.x, so we have
    that in our favor.
  - When both IUKs are present at the same time (i.e. before rebooting
    after having applied the upgrade) the disk usage is 1760 MB + 682
    MB = 2442 MB.
  - After the old IUK is purged the disk usage is down to 2442 MB -
    678 MB = 1764 MB.
* Memory needs: while performing the upgrade, 2x the size of the IUK
  of memory is needed, so 2*682 MB = 1364 MB. On a system with 2 GB of
  memory, a fresh boot of Tails 3.5 (without networking to limit the
  amount of processes autostarting) has 1255 MB of free memory
  (according to `check_free_memory()`'s calculation in
  `config/chroot_local-includes/usr/local/bin/tails-upgrade-frontend-wrapper`)
  so the upgrade would fail. This is a regression for users with 2 GB
  memory: for all of 2.x and 3.x, all IUKs have been under 400 MB,
  which would work fine with 2 GB of memory. If that's a blocker, then
  we have to solve it by changing the format of the IUKs
  ([[!tails_ticket 6876]]); we should coordinate this with other changes
  that will break automated upgrades from Tails N to N+1, such as
  Tails 4.0 and the migration to overlayfs ([[!tails_ticket 9373]]) .
* Bandwidth needs of the RM. Uploading 10 GB of IUKs can be a pain for
  some of us, but that can easily be solved by making it possible to
  generate IUKs on lizard (and then compare them with the ones you
  generated locally. Thanks reproducibility!).

## Support upgrading very old Tails

We probably do not want to support upgrading very old (e.g. from six
months ago) Tails installations because our signing key might have
been updated in the meantime, and/or some critical update to Tails
Upgrader (e.g. UDF version). So unless we want to spend a lot of
resources on solving these problems, which roughly translates to "you
can take long breaks from Tails", let's guarantee our current
"you can at most skip one *planned* release" promise, and
optimistically support older versions whose key and Tails Upgrader are
still ok.

## Download requirements

tl;dr: for 2.x the new scheme would have required downloading 50% more
data than the original scheme.

### 2.x stats for original IUK scheme

Assumptions:

 * a 4 GB Tails system partition.

It would look like this:

    1082M   tails-i386-2.0.iso
    153M    Tails_i386_2.0_to_2.0.1.iuk
    315M    Tails_i386_2.0.1_to_2.2.iuk
    104M    Tails_i386_2.2_to_2.2.1.iuk
    258M    Tails_i386_2.2.1_to_2.3.iuk
    207M    Tails_i386_2.3_to_2.4.iuk
    324M    Tails_i386_2.4_to_2.5.iuk
    363M    Tails_i386_2.5_to_2.6.iuk
    328M    Tails_i386_2.6_to_2.7.iuk
    197M    Tails_i386_2.7_to_2.7.1.iuk
    176M    Tails_i386_2.7.1_to_2.9.1.iuk
    1153M   tails-i386-2.10.iso
    300M    Tails_i386_2.10_to_2.11.iuk
    296M    Tails_i386_2.11_to_2.12.iuk

Total: 5256 MB

### 2.x stats for 1BigIUK scheme

Assumptions:

 * a 4 GB Tails system partition.
 * original installing 2.0 creates the largest IUKs.

It would look like this:

    1082M   tails-i386-2.0.iso
    153M    Tails_i386_2.0_to_2.0.1.iuk
    388M    Tails_i386_2.0_to_2.2.iuk
    395M    Tails_i386_2.0_to_2.2.1.iuk
    434M    Tails_i386_2.0_to_2.3.iuk
    462M    Tails_i386_2.0_to_2.4.iuk
    515M    Tails_i386_2.0_to_2.5.iuk
    590M    Tails_i386_2.0_to_2.6.iuk
    597M    Tails_i386_2.0_to_2.7.iuk
    621M    Tails_i386_2.0_to_2.7.1.iuk
    622M    Tails_i386_2.0_to_2.9.1.iuk
    678M    Tails_i386_2.0_to_2.10.iuk
    682M    Tails_i386_2.0_to_2.11.iuk
    634M    Tails_i386_2.0_to_2.12.iuk

Total: 7853 MB

### FWIW, IUK sizes for 3.x

    200M    Tails_amd64_3.0_to_3.1.iuk
    278M    Tails_amd64_3.1_to_3.2.iuk
    260M    Tails_amd64_3.2_to_3.3.iuk
    177M    Tails_amd64_3.3_to_3.4.iuk
    213M    Tails_amd64_3.4_to_3.5.iuk