summaryrefslogtreecommitdiffstats
path: root/wiki/src/todo/usb_install_and_upgrade.mdwn
blob: 500d775aacedc49b1268eb0b876b845f6c0daeb1 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
[[!meta title="USB install and upgrade"]]

[[!toc levels=3]]

Current state of things
=======================

Tails [[ships hybrid ISO images|contribute/design/hybrid_ISO]]. dd'ing
such a file to a USB stick produces a bootable Tails system and is
[[well documented|support/installing_onto_a_usb_stick]], but it
requires unfriendly command-line usage and overwrites any previously
existing partition table.

The current state of things implies two distasteful facts:

1. A given USB stick currently must be dedicated to Tails only;
   considering the ever-growing size of such devices, the amount of
   wasted bytes is not a minor one.
2. A given USB stick cannot be used to host both a Tails system *and*
   a persistent storage volume, which makes it hard to provide
   friendly integration of [[upcoming persistence
   features|todo/persistence]] for end-user.

Such user unfriendliness conflicts with Tails [[specified
goals|contribute/design]] and must be fixed.

Specification
=============

One should be able to easily install Tails to a USB storage device:

* by cloning an existing Tails system that is running from CD or USB;
* running a non-Tails operating system, from a downloaded ISO image.

The installation target may be an existing partition whose content
must be left untouched. The choice between possible destination
devices or partitions must be graphically proposed amongst the
available removable storage devices.

Moreover, at installation time, one should be allowed to setup an
encrypted storage space that can be used by the [[upcoming persistence
features|todo/persistence]]; earliest versions of the installation
program must at least leave room for it so that a later version can be
make use of it. Whether this storage space should better be a second
encrypted partition or a file-backed encrypted FS shall be researched
and decided upon. If the partition way is chosen, the installer
software must have partitioning capabilities.

One should also be able to easily and non-destructively upgrade a
given Tails system installed on a USB storage device.
Non-destructively means not touching any partition or file other than
the ones managed by the Tails installation system. Proceeding this way
is needed to make it possible using a single USB stick for the Tails
system *and* some persistent storage space.

Research
========

[[!tag todo/research]]

Frontend UI
-----------

### Overview

Unfortunately, no single existing piece of software satisfies the
above specification. It seems like everyone writes her own USB
installer; after the following round-up has been made, this is no
surprise at all: since such tools are designed to be dead easy for the
non-technical end-user, every author selects the few features that are
the most important ones for her intended audience, and generally
writes a very ad-hoc piece of software. Therefore, every Live system
project whose needs are slightly different from the others needs to
write her own USB installer, possibly reinventing more or less wheels
along the road.

We need to see if we can turn one of those tools into something that
satisfies our specification; sadly, proceeding this way *without
forking the existing tool* would probably need more initial
development time than writing a new installer from scratch; we are not
even sure any upstream author of such a software (good enough to be a
possible viable basis) would want to make it more generic and maintain
it this way; on the other hand, if we avoid forking or writing from
scratch, we also avoid maintaining a Tails-specific tool on the long
run.

### Existing tools

[[!tag todo/test]]

(Testing all these is probably [[!taglink todo/easy]].)

Wikipedia has a [[!wikipedia List of tools to create Live USB systems]]
we should look at.

* `live-installer` udeb and its `live-installer-launcher` desktop
  companion allows to run the Debian Installer from the bootloader;
  paraphrasing the manual: the "Live" Debian Installer flavour will
  proceed in an identical fashion to the normal installation but at
  the actual package installation stage, instead of using
  `debootstrap` to fetch and install packages, the "live" filesystem
  image is copied to the target. After this stage, the Debian
  Installer continues as normal, installing and configuring items such
  as bootloaders and local users, etc. This must be tested.

  > Not usable without a lot of work: there is way too many
  > questions asked, the graphical interface overlaps the
  > panels (hence unreachable buttons), it tries to create
  > a new user account, etc. Forget it, even if it was nice
  > to have partman handy to configure partitions and
  > bootloaders.

* `live-boot`'s `todisk=DEVICE` boot parameter could be used to copy
  the entire read-only media to the specified device; the process
  could then be:

  1. boot from CD
  2. use some documentation or GUI to partition the USB stick and take
     note of its device name
  3. reboot; at syslinux prompt add the `todisk=DEVICE` parameter
     using the previously noted device name; then let the system boot
     once from the USB stick, and use some documentation or GUI to
     install a bootloader on the USB stick. This step might be
     automated in live-boot.
  4. reboot and eject the CD: the system should now boot from USB.

  This only works for the "clone existing system" usecase, though.

* Incognito's `root_overlay/usr/sbin/create-usb` seems quite nice, as
  it allows using an existing fat32 partition.
   - no partition capabilities

* Liberté Linux' `burn-usb` creates a FAT32 partition for data and
  an ext2 partition where it copies the system to. It installs Grub on
  the USB stick.

* Ubuntu's [usb-creator](https://launchpad.net/usb-creator)
   - supposed to replicate exactly the CD to USB
   - does not seem to recognize non-Ubuntu CDs (0.1.16 crashes on
     Lenny, 0.1.10 does not display them in the GUI even if it the
     console output proves that it takes good note of their presence )
   - how does it handle partitions?

* [UNetbootin](http://unetbootin.sourceforge.net/)
   - works on Windows and GNU/Linux
   - has a plugin architecture that only allows rebranding and
     pre-selection of the available distributions list
   - included in squeeze and sid, easily backported to Lenny
   - works like a charm to install an ISO image to USB, but the
     syslinux menu has the unetbootin logo instead of Debian's one;
     the installed syslinux defaults to the first menu entry (in
     Tails case, Arabic); are the other bits of syslinux menu (Debian
     Live help and so on) here?
   - part of Debian
   - non-destructive install: does not format the chosen partition
   - no support for partitioning nor persistent storage
   - works from a running Live CD; specify /dev/cdrom as ISO diskimage
   - how would upgrading work?

* [Ubuntu LiveUSB](http://klik.atekon.de/liveusb/)
   - (a bit too) simplistic
   - dedicated to "create a bootable Live USB medium from a running
     Ubuntu Live CD"
   - depending on sources, it partitions USB stick with one single
     800MB partition or one partition using the whole device; in any
     case, it unconditionally overwrites the device's content
   - able to upgrade without re-partitioning the device?
   - [launchpad](https://launchpad.net/liveusb)

* aptosid's `aptosid-on-a-stick` needs an input ISO (check again), and
  lacks partitioning capabilities, but it still deserves being looked
  at more thoroughly; see the
  [manual](http://manual.aptosid.com/en/hd-install-opts-en.htm#usb-hd).

* Fedora's `livecd-iso-to-disk`:
   - [manual](http://fedoraproject.org/wiki/FedoraLiveCD/USBHowTo)
   - [Gitweb](http://git.fedorahosted.org/git/?p=livecd)
   - cleanly and defensively written in Bash
   - supports creating a file-backed filesystem for `$HOME`,
     LUKS-encrypted by default, in the same partition where the Live
     system is installed
   - uses parted

* Fedora's [liveusb-creator](https://fedorahosted.org/liveusb-creator/):
   - works on Windows and GNU/Linux
   - "non-destructive install", which means?
   - supports FAT and EXT filesystems
   - persistent storage creation: file-backed filesystem, no
     partitioning capabilities
   - actively developed (as of early 2011)
   - written in Python
   - input: ISO file or real CD
   - output: extracts ISO's content to the partition
   - does not use `livecd-iso-to-disk`
   - needs to be slightly adapted/extended to support non-Fedora Live
     systems, e.g. some directory names must be made configurable
   - sidux ships a modified version:
      - [.deb](http://sidux.office-vienna.at/sidux/debian/pool/main/l/liveusb-creator/),
      - [source](http://w3you.com/liveusb-creator/liveusb-creator.htm)

* [Mandriva Seed](http://git.mandriva.com/?p=projects/mandriva-seed.git;a=summary)
   - supports Windows and GNU/Linux
   - fork of Fedora's liveusb-creator; what are its enhancements?

* [RIP’s mkusb](http://rip.7bf.de/current/mkusb.sh) and its
  [README](http://rip.7bf.de/current/mkusb.txt)

* SUSE Studio Image Writer([[!wikipedia SUSE Studio ImageWriter]],
  [KIWI homepage](http://kiwi.berlios.de/), [source
  code](http://git.berlios.de/cgi-bin/gitweb.cgi?p=kiwi;a=tree;f=tools/burner))
   - supports Windows and GNU/Linux
   - persistent storage creation - that is?

* [makebootfat](http://advancemame.sourceforge.net/boot-readme.html):
  command line utility able to create bootable USB disks
  using the FAT filesystem and syslinux
   - available in Debian
   - able to autodetect/partition/format/populate the USB disk in a
     single step without any user interaction
   - able to create disk images which are compatible with all the
     three standards USB-FDD, USB-HDD and USB-ZIP at the same time
   - could be used to do most of the low-level work

* [FUSBi](http://aligunduz.org/FUSBi/)
   - modified from UNetbootin; what are its enhancements?
   - supports Windows and GNU/Linux
   - no release in 2009 nor 2010

#### Dismissed

* Moblin's [Image
  Writer](http://git.moblin.org/cgit.cgi/moblin-image-creator/plain/image-writer)
   - written in Python
   - simple dd wrapper with proper sanity checks

* [usb-imagewriter](https://launchpad.net/usb-imagewriter)
   - seems like a clone of the good old rawwrite with no more feature
     at all

* <http://wubi-installer.org/>: Windows installer for Ubuntu.

Booting methods
---------------

### Overview

Tails [[ships hybrid ISO images|contribute/design/hybrid_ISO]]. dd'ing
such a file to a USB stick produces a bootable Tails system but it
overwrites any previously existing partition table. This method is
therefore not suitable for us since it does not satisfy the
"non-destructive upgrade process" requirement.

Two alternatives booting methods have been investigated:

1. **Boot ISO file**: Setup a bootloader on the USB stick that is able
   to boot an ISO image dropped into the right directory of it. Even
   early boot files (kernel, initramfs) are loaded from *inside* the
   ISO file. The bootloader installed on the USB stick can either
   directly boot the kernel from inside the ISO file or chainload the
   ISO image's bootloader. In the first case, until the
   [[todo/boot_menu]] task is completed, the installed boot loader
   must present a boot menu that at least allows the user to choose
   her preferred language.

2. **Copy ISO's content**: Copy needed files (mainly the kernel,
   initramfs and SquashFS image that live in the `live/` directory)
   from the ISO filesystem to the destination partition, then setup a
   bootloader that uses files in that directory. The end result is
   more or less equivalent to dd'ing the ISO image to the USB stick,
   but achieved with a non-destructive process.

### Grub2

#### Grub2's loopback support

Debian Live now supports Grub2 and `fromiso=`. The grub.cfg that is
generated inside the binary image could be copied, adapted and used to
(loopback) boot an ISO dropped on a USB stick.

Here is how to test this.

0. Build a Debian Live iso or iso-hybrid image with a working
   `live-boot` installed in it, i.e. at least 2.0.12 (warning: Tails
   0.6 series ship a buggy version).
1. Create a ext2 filesystem on a USB stick partition.
2. Copy a Tails ISO file **at the root** of the USB stick partition.
3. Run `ln -s . boot` at the root of the (mounted) USB stick partition.
4. Install Grub2 on the USB stick MBR, using the USB stick partition
   as `--root-directory`.
5. Create a `grub.cfg` file in there and try a menu entry like this:

          menuentry 'test' --class debian --class gnu-linux --class gnu --class os {
                insmod part_msdos
                insmod ext2
                insmod loopback
                insmod iso9660
                set root='(hd0,msdos1)'
                search --no-floppy --fs-uuid --set b51975cd-c5ef-4b32-996a-2c8206516917
                loopback loop /binary.iso
                linux   (loop)/live/vmlinuz boot=live fromiso=/dev/sda1/binary.iso config
                initrd  (loop)/live/initrd.img
          }

Details:

* `insmod ext2` is because the partition that hosts Grub2 and the ISO
  is a ext2 one
* `(hd0,msdos1)` and the search UUID match this partition as well,
  from Grub2 point-of-view
* `/dev/sda1` is a path to this partition block device from inside the
  Live initrd
* `/binary.iso` is the path to the ISO file, relative to the root of
  the partition
* `/live/vmlinuz1` and `/live/initrd1.img` are respectively the path
  to the kernel and ramdisk relative to the root of the ISO filesystem
* `boot=live` enables `live-boot` and thus `fromiso=`
* `config` enables `live-config`

This might fails at kernel load time with following error:

    error: invalid magic number

In that case you probably need a newer Grub2 (see [[!debbug 543924]]
for details).

**Warning!**

1. This breaks the [[emergency shutdown|contribute/design/memory_erasure]]
   on media removal.
2. This is by no means a supported installation method *yet*. E.g. the
   kernel parameters shall be manually updated when replacing the ISO
   with a newer one.

#### Security matters

In the above example, the Grub2 configuration file must contain
information about the partition where the ISO file is stored.

Doing this is feasible at install time, and ensures the ISO will be
loaded from the right partition, rather than from a rogue one on the
host system's hard-disk.

Other Live systems that don't face the same threat model as Tails tend
to prefer using automagic ISO loopback support so that Grub2 itself
finds a partition where a properly named ISO is stored, e.g. using
[GRML's patches against
live-boot](http://git.grml.org/?p=live-boot-grml.git;a=tree;f=debian/patches)
that add the `findiso=` boot parameter (by the way, does it respect
`live-media=removable` ?); here is how they boot things up:

- GRML (`grml-rescueboot` flavor) [now supports booting using Grub2
  loopback](http://michael-prokop.at/blog/2011/01/07/booting-iso-images-from-within-grub2/)
- comments of
  http://blog.cyphermox.net/2010/10/booting-to-iso-images-from-usb-key.html
- Super Grub Disk's
  [loopback.cfg](http://www.supergrubdisk.org/wiki/Loopback.cfg) might
  be [going
  upstream](http://lists.gnu.org/archive/html/grub-devel/2010-09/msg00080.html).

#### Grub2 combined with syslinux' memdisk

[[!tag todo/research]]

This is used by some in combination with Grub2, in place of its
loopback support. The [memdisk
page](http://syslinux.zytor.com/wiki/index.php/MEMDISK) on the
syslinux wiki tells the whole low-level story and more.

Resources:

- GRML (`grub-imageboot` flavor) [now supports booting using
  Grub2+memdisk](http://michael-prokop.at/blog/2011/01/07/booting-iso-images-from-within-grub2/);
  according to this blog post, support for doing this is in live-boot
  2.0.14-1. The memdiskfind/phram/mtdblock approach this blog post
  talks of would ease cloning an existing Tails system, since it is
  supposed to provide a `/dev/mtdblock0` device, which should be the
  .ISO image. See the [details on the memdisk wiki
  page](http://syslinux.zytor.com/wiki/index.php/MEMDISK#-_memdiskfind_in_combination_with_phram_and_mtdblock).
- a [blog
  post](http://blog.snow-crash.org/2010/11/cool-things-with-grub2-and-syslinux---booting-floppy-and-iso-images.html)
  with a snippet for `grub.d` to automate booting ISO images from
  Grub2, using syslinux' memdisk

### extlinux and memdisk

According to Philip Hands, who proposed this solution on [[!debbug
534887]], it is possible to use extlinux merely to chainload the
image's own bootloader.

Partitioning
------------

Various scripts, such as [Benjamin FOURTICQ's
one](http://www.mail-archive.com/debian-live@lists.debian.org/msg01832.html),
aim at adding a persistent home-rw partition to a Debian Live disk
image, generally using losetup and parted. They don't suit Tails
specification as they depend on knowing in advance the size of the
persistent volume the user wants to setup and obviously cannot setup
encryption on it in place of the user.

On the other hand,
[udisks](http://www.freedesktop.org/wiki/Software/udisks) provides a
D-Bus interface that makes it easy to programmatically partition a
storage device and create filesystems in there without messing with
low-level details; its LUKS encryption support probably is the killer
feature that makes it the perfect library for the job.