[[!meta title="Endless upgrades"]]
Currently automatic upgrades have the limitation that only some N
number of them can be performed in a row before the system partition
runs out of space and automatic upgrades cannot be done further, but a
full, manual upgrade is needed.
Automatic upgrades should be possible forever. This would:
- Improve UX as currently full upgrades are one of the most complicated and
long scenario, requiring at least 2 USB sticks and 2 restarts.
- Improve security by eliminating the need for recurring verifications
of ISO images and reducing cross-contamination by limiting the
number of times you might want to clone or rely on another operating
system for downloading or upgrading.
# Different approaches to endless upgrades
While examining these approaches, let's forget about IUKs for a while,
and only consider full automatic upgrades ([[!tails_ticket 7499]]):
## Extract upgrade from ISO
Keeping our current partition layout and filesystems, a full upgrade of
a running system can be done by extracting the required parts from the
new version's .iso. Here we'd need at least 3 times the size of an
1. one for the old, running installation
2. one for the new ISO
3. one for the extracted contents of the new ISO
We would unlink the old squashfs, kernel, initrd etc. then copying in
their place new version's ones, so the new version will run next time
### Disk images instead of ISOs
Note that if we switch to using disk images as the primary way to
install Tails ([[!tails_ticket 8550]], comment 9) we can similarly
extract the needed parts just like above: replace "(ISO|.iso)" with
"disk image" and it still works.
## Streaming decompression
An improved version of the above idea would be to serve "upgrade
packs" using some compression method friendly to streaming
decompression, so the need to download a the full archive (ISO or disk
image above) is avoided, which reduces the amount of temporary disk
space needed from 3 to 2 times the size of an installation when
performing the upgrade.
Grub and syslinux supports booting directly from an ISO so we could
switch to a strategy where Tails USB/SD installation means that we
dump the ISO image into the system partition, and configure the
bootloader to boot from it. If we make the system partition large
enough to store two ISOs (= 2 times the size of an installation, when
comparing o the above methods) we can keep the older, running version
in place while downloading the new one, and then safely switch to the
new one at next boot. Since the old version is kept, there's a
fallback in case the new version doesn't boot. Older versions can be
cleaned up at the next upgrade.
Depending on how Grub/syslinux does the ISOBoot, this may make all
running Tails instances more consistent, i.e. we now do not have the
three cases of what type of medium Tails has booted from: DVD; USB
drive; or SD card -- in all cases we have a ISO9660 filesystem. OTOH,
the magic employed for the ISOBoot may hide too much about underlying
real device, which we still probably will need information about.
Some more details (some perhaps wrong) can be read in comment 16 on
## Apply upgrade in initramfs
By storing the ISO (or disk image) that the upgrade is gonna be
extracted from on a scratch partition, and then apply the upgrade in
initramfs on next boot, we only need 2 times the size of an
installation in total, and can safely download the ISO to the scratch
partition while keeping the system partition read-only.
Ideally, let's also save the files used to authenticate the upgrade on
the scratch partition, and make check the authenticity during
initramfs, before starting the upgrade.
Also, with the scratch partition we will never have to mount it
read-write outside of initramfs, it can be made read-only in a
slightly stronger sense on the block dev level. This is not really a
security feature, but more a foolproof things, but why not?
## How others do i.e. non-NIH solutions
This survey will be updated in a while with [[!tails_ticket 15277]].
* yocto project's
[comparison of system update mechanisms](https://wiki.yoctoproject.org/wiki/System_Update)
and A/B partition setups, e.g.
([example config file](https://github.com/rauc/rauc/blob/master/contrib/grub.conf))
- systemd integration and D-Bus API
- super flexible
- no support for binary delta updates yet, but there's
a [RFC for casync support](https://github.com/rauc/rauc/pull/217)
which downloads only missing bits and downloads/applies with
streaming (no intermediate download of images)
- ITP: [[!debbug 916210]]
* [Qt's Over-The-Air Update](https://doc.qt.io/QtOTA/) aka. QtOTA,
that uses OSTree
* XXX: other OSTree based designs/implementations?
* XXX: other [casync](http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html) ([website](https://github.com/systemd/casync))
* XXX: how efficient is casync with SquashFS? in other words, do we
need to unpack the rootfs on the target system partition?
* Chrome OS keeps the running system partition read-only, has _two_
system partitions, and the update is applied to the non-running one.
This has nice additional properties like allowing one to rollback if
the upgraded system doesn't start properly. More information about
their design and implementation:
- [disk format](https://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format)
- [file system and autoupdate system](https://www.chromium.org/chromium-os/chromiumos-design-docs/filesystem-autoupdate)
- [Android's A/B System Updates](https://source.android.com/devices/tech/ota/ab_updates.html)
* [Endless ostree builder](https://github.com/cosimoc/deb-ostree-builder)
* [[!debbug 824649 desc=" ostree: document how to prepare and update a .deb-based system root"]]
### Not suitable
(but could be good sources of inspiration!)
[OSTree based upgrader](https://github.com/endlessm/eos-updater)
- supports separate poll/fetch/apply operations (so presumably we
could make the relevant partition read-write only after fetching,
as long a we can fetch to a tmpfs)
- daemon + D-Bus API
- systemd integration
- supports Flatpak
- likely too many EndlessOS-specific assumptions… and written in
C so difficult to adapt to our needs
* [mender](https://mender.io/): downloads full update
* [swabic's swupdate](https://sbabic.github.io/swupdate/): downloads
full update; binary delta updates are
[on the roadmap](https://sbabic.github.io/swupdate/roadmap.html#binary-delta-updates)
# Auxiliary ideas
## Read-write system partition while the network is enabled
Currently we keep the system partition read-only as long as the
network is enabled. E.g. we download IUKs into the tmpfs, and when
it's finished we disable the network and only then remount the system
partition read-write to apply the upgrade.
This is problematic for full upgrades, since we cannot assume that
users' systems has over 1 GiB of free RAM available but need to
download it to disk, but cannot use the system partition. In the
"extract upgrade from ISO/Disk image" case we can simply introduce a
scratch partition where the temporary download is stored without loss
of disk usage efficiency (we still need 3 times an installation). But
for the ISOBoot and streaming decompression cases we'd then go from 2
times to 3 times the size of an installation.
It might be worth having a discussion about whether preventing the
network to be used while the system partition is mounted read-write
actually brings us any security. After all, if the system is
compromised while downloading the upgrade, it will still be
compromised when the download is finished and the system partition is
remounted read-write, and it can infect that part then.
Note that with A/B system updates this discussion is moot: the
bootloader configuration update is the only part of the process that
requires read-write access.
## What about IUKs?
If we want to support IUKs at the same time as full upgrades, we need
to allocate even more space to the system partition. Also, we need to
carefully consider the UX part of this.
For instance, we could always give users the choice to do a full or
incremental upgrade unless installing the incremental upgrade will
result in too little disk space remaining for a full upgrade to be
done *next* time -- in this case we only allow full upgrades, to
prevent users from locking themselves in a situation where they cannot
upgrade further. When both options are available, we could give users
the suggestion that they should do a full upgrade if they currently
have a lot of time and/or a fast connection.
## Self-contained verification
An interesting option for Tails installation verification
([[!tails_ticket 7496]]) is to contain all the bits needed to verify
an installation's integrity. For instance, in the ISOBoot case we'd
include the .sig for the ISO on as well as the .iso, so an external
verifier simply can verify the .sig instead of needing an external
known-good .iso to compare with.
The ISOBoot case is the simplest, probably, as the other methods would
require signed manifests of files with the expected hashes (or
similar), which of course is not such a big deal to provide.
The ISOBoot case would still require a manifest of files we extract
from the ISO (bootloader code and configuration), but indeed that
manifest could be in the ISO itself, and thus its authenticity and
integrity would be signed/verified at the same time as the ISO.
# Temporary stop gap: stack one single SquashFS diff when upgrading
(This idea was originally conceived in the comments of [[!tails_ticket
Without any major redesign of our current incremental upgrade system
(i.e. comparatively cheap) we can allow users upgrade only through
IUKs for much longer, probably throughout a whole Tails series,
e.g. all of Tails 2.x, which we'll use for our examples below since we
moved to 3.x already so we have all data we need to test this idea.
## Basic idea
### Inside Tails
* Tails Upgrader fetches the UDF for the version of Tails that was
originally installed (instead of the version of the running Tails),
* We remove any previous IUK (and reference to it in `Tails.module`)
when a new IUK is installed.
### Release management
When we release version M.N, then, for all previous versions M.X since
M.0, we generate a M.X_to_M.N IUK.
(Actually, it is not necessary to choose the first release of a series
(like M.0) as the "synchronization point". We could add another one
half way through the series, or make every tenth release a
synchronization point, or we could do it when when the IUK simply
grows too big. We have lots of options to balance this according to
our needs, but the first release of a series seems like a technically
reasonable choice according to the data we have for the 2.x series
(see below), but also nice from the user's PoV if the only time a
manual upgrade is needed is when upgrading between series.)
1. I install Tails 2.2 on a USB stick.
2. Later when Tails 2.3 is released I upgrade to it via the 2.2_to_2.3
IUK (i.e. identical to our original scheme).
3. When Tails 2.4 is released I upgrade to it via the 2.2_to_2.4 IUK
(whereas in our original scheme we would stack 2.3_to_2.4 on top of
2.2_to_2.3) and remove the 2.2_to_2.3 IUK (whereas in our original scheme
we never remove IUKs).
Notice how step 3 can be repeated for each new release, so that we in
the end reach the final release of the series, Tails 2.12. And notice
how we in step 1 could have installed any 2.x release.
## Problems due to IUK size
We'll be dealing with larger IUKs (because generally more and more
changes accumulate between different Tails versions) and only have one
present at a time, with the exception being while we are installing a
new IUK (then two will be present). Let's look at the IUK sizes
towards the end of the 2.x series as if the last manual install was
for Tails 2.0, which is the excepted worst case:
The IUK size is involved in at least four concerns:
* Disk space needs on mirrors: for 2.x there were 14 non-rc/beta/alpha
Tails releases, so there would be 13 IUKs for the last release
(2.12). So using the worst-case as the average case we get the upper
bound of the space needs on each mirror: 14*682 MB ~= 10 GB.
* Disk space needs on the Tails system partition: we need to be able
to store two large IUKs. Apparently the worst case for 2.x is the
2.10 to 2.11 upgrade if 2.0 was the version originally installed:
- The 2.0 installation itself uses 1082 MB, so upgraded to 2.10 the
total usage is 1082 MB + 678 MB = 1760 MB.
- Due to the Upgrader requiring 3x the size of the IUK on the Tails
system partition, the actual requirement is 1760 MB + 3*682 MB =
3806 MB of free disk space. That wouldn't have worked during the
2.x times, when the system partition was 2.5 GB, and it barely
would work with our current 4.0 GB partition. OTOH, during the 2.x
series we shipped two kernels, while we don't in 3.x, so we have
that in our favor.
- When both IUKs are present at the same time (i.e. before rebooting
after having applied the upgrade) the disk usage is 1760 MB + 682
MB = 2442 MB.
- After the old IUK is purged the disk usage is down to 2442 MB -
678 MB = 1764 MB.
* Memory needs: while performing the upgrade, 2x the size of the IUK
of memory is needed, so 2*682 MB = 1364 MB. On a system with 2 GB of
memory, a fresh boot of Tails 3.5 (without networking to limit the
amount of processes autostarting) has 1255 MB of free memory
(according to `check_free_memory()`'s calculation in
so the upgrade would fail. This is a regression for users with 2 GB
memory: for all of 2.x and 3.x, all IUKs have been under 400 MB,
which would work fine with 2 GB of memory. If that's a blocker, then
we have to solve it by changing the format of the IUKs
([[!tails_ticket 6876]]); we should coordinate this with other changes
that will break automated upgrades from Tails N to N+1, such as
Tails 4.0 and the migration to overlayfs ([[!tails_ticket 9373]]) .
* Bandwidth needs of the RM. Uploading 10 GB of IUKs can be a pain for
some of us, but that can easily be solved by making it possible to
generate IUKs on lizard (and then compare them with the ones you
generated locally. Thanks reproducibility!).
## Support upgrading very old Tails
We probably do not want to support upgrading very old (e.g. from six
months ago) Tails installations because our signing key might have
been updated in the meantime, and/or some critical update to Tails
Upgrader (e.g. UDF version). So unless we want to spend a lot of
resources on solving these problems, which roughly translates to "you
can take long breaks from Tails", let's guarantee our current
"you can at most skip one *planned* release" promise, and
optimistically support older versions whose key and Tails Upgrader are
## Download requirements
tl;dr: for 2.x the new scheme would have required downloading 50% more
data than the original scheme.
### 2.x stats for original IUK scheme
* a 4 GB Tails system partition.
It would look like this:
Total: 5256 MB
### 2.x stats for 1BigIUK scheme
* a 4 GB Tails system partition.
* original installing 2.0 creates the largest IUKs.
It would look like this:
Total: 7853 MB
### FWIW, IUK sizes for 3.x