summaryrefslogtreecommitdiffstats
path: root/wiki/src/contribute/APT_repository.mdwn
blob: ec87c2dc790664c4e10686511529ccc837230bbe (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
[[!toc levels=2]]

To avoid cluttering our main Git repository with Debian source and
binary packages, we have set an APT repository up.

[[!toc levels=2]]

Overview
========

We use one single APT repository hosting multiple *suites*:

* We have a (read-only) suite for every past release: `0.9`,
  `0.10.1`, etc.
* We have a suite for each *main* branch: `stable`, `testing`,
  `devel`, `feature-jessie`
* We have an overlay suite for each *topic* branch: `bugfix/*`,
  `feature/*`, etc.
  **Note**: the APT suite corresponding to a given Git topic
  branch contains *only* the packages this branch adds to the tag or
  *main* branch it diverged from. Think of it as an overlay.
* We also have a less formal `unstable` suite, that should not be used
  by any Tails git branch; it can be used as hosting space for other
  packaging work we might do, e.g. acting as upstream or
  Debian maintainers.
* We also have a `builder-wheezy` suite, used to provide additional
  packages needed on a Wheezy system to build Tails.

The suite(s) to use as sources for APT, during the build and inside
the resulting system, are determined by the content of the
`config/base_branch` and `config/APT_overlays.d/*` files. See details in
the *Build system* section below.

We manage our APT repository with
[reprepro](http://mirrorer.alioth.debian.org/).

The Puppet modules used to manage this part of our infrastructure are
listed on our [[contribute/Git]] page.

Basically, a cronjob fetches and scans the Tails Git repository every
few minutes, detects new branches, and accordingly:

- generates `conf/distributions`
- generates `conf/incoming`
- create new suites in the APT repository

Build system
============

The Tails ISO build system dynamically adds APT sources that will be
used during the build, and inside the resulting ISO itself.

If the last version in `debian/changelog` was released already (i.e.
a matching tag exists), then the build system adds the suite
corresponding to this release (e.g. `1.5` or `3.0`), and that's all.

Else, it adds:

* one APT source for the base branch of the one being built, as found
  in `config/base_branch`;
* one APT source for each suite listed in
  `config/APT_overlays.d/*`; note that only the name of such
  files matters, and their content is ignored.

In practice, `config/APT_overlays.d/` contains:

* for a topic branch:
  - if needed, a file that is named like the branch's own overlay APT
    suite; e.g. for the `bugfix/12345-whatever` branch, it would be
    called `config/APT_overlays.d/bugfix-12345-whatever`
  - any file representing APT suites that came from merging its base
    branch into this topic branch, that is:
* for a base branch (`stable`, `testing`, `devel` or
  `feature/jessie`): a file for each additional, overlay APT suite that
  came from topic branches that ship Debian packages and were merged
  into this base branch since last time it was used to prepare
  a release.

The code that implements this is [[!tails_gitweb
auto/scripts/tails-custom-apt-sources]]. It has [[!tails_gitweb
features/build.feature desc="automated tests"]].

At release time, the release manager:

1. merges into the release branch's APT suite all APT overlay
   suites found in `config/APT_overlays.d/`;
2. empties `config/APT_overlays.d/` in the release branch;
3. merges the release branch into other base branches as needed, and
   ensures that all resulting `config/APT_overlays.d/`:s make sense.

Note that a branch like `feature/jessie` needs to be a base branch: we want to be
able to work on topic branches forked off `feature/jessie`.

SSH access
==========

One must configure their SSH client to connect to the APT server:

	Host incoming.deb.tails.boum.org
		Port 3003

HTTP access
===========

This is the http:// public APT repository used at Tails
build time. The `tails::reprepro` Puppet class sets nginx up to
serve that.

Workflow
========

Creating a new branch
---------------------

Push your branch to Git and wait a few minutes for the new APT suite
to appear.

Then you probably want to tell the build system, via
`config/APT_overlays.d/` (documented above), that this new APT suite
must be used when building that new branch.

Importing a new package
-----------------------

### Building a package

Make sure the `Distribution:` field in your `.changes` file matches
the suite you want the package to land in (e.g.
pass `--changes-option=-DDistribution=feature-torbrowser` to
pdebuild's `--debbuildopts`).

Make sure to have the `.changes` file include the original source
archive (`.orig.tar.{gz,bz2,xz}`) if it is not already in our APT
repository; this can be done by passing `-sa` to pdebuild's
`--debbuildopts`.

### Configuring an upload tool

#### Configuring dupload

Add this configuration snippet to your `dupload` configuration:

	$config::cfg{'tails'} = {
	        fqdn => "incoming.deb.tails.boum.org",
	        method => "scp",
	        login => "reprepro",
	        incoming => "/srv/reprepro/incoming/",
	        dinstall_runs => 1,
	};

#### Configuring dput

Add this to `.dput.cf`:

    [tails]
    fqdn            = incoming.deb.tails.boum.org
    method          = scp
    login           = reprepro
    incoming        = /srv/reprepro/incoming/
    run_dinstall    = 0


### Uploading and importing process

Carefully prepare and build your package. Usual precautions, (Lintian
etc.) apply.

Carefully check the `.changes` file (especially the `Distribution`
control field, and the included files list; the former can be fixed
with the `changestool(1)` command, from [[!debpkg reprepro]]).

Sign the `.changes` file with a key that is in the uploaders list:

	$ debsign $CHANGES_FILE

Upload the files to the incoming queue:

	$ dupload --to tails $CHANGES_FILE

reprepro will automatically notice the new files and import them into
the suite specified in your `.changes` file.

Check the result:

	$ ssh reprepro@incoming.deb.tails.boum.org reprepro list $SUITE $PACKAGENAME

<a id="workflow-merge-main-branch"></a>

Merging a main branch
----------------------

When a Git *main* branch (`devel`, `testing`, `stable`,
`feature/jessie`) is merged into another *main* branch, the corresponding
operation must be done on the APT suites.

1. Save the list of packages currently present in the APT suite we
   want to merge *into*, e.g. `reprepro list devel`.

2. Make sure you are not going to overwrite newer packages with
   older ones (hint: use the `tails-diff-suites` script).

3. Merge the APT suites:

   1. Set some environment variables:

            # the branch you want to merge
            SRC=stable
            # the branch you want to merge _into_
            DST=devel

   2. Merge in Git and APT:

            git checkout "$DST" && \
            git merge "$SRC" && \
            ssh reprepro@incoming.deb.tails.boum.org \
                 tails-merge-suite "$SRC" "$DST"

   3. Restore the `config/base_branch` if needed:

            echo "${DST}" > config/base_branch && \
            git commit config/base_branch -m "Restore ${DST}'s base branch." || :

   4. Push:

            git push origin "${DST}:${DST}"

4. Make sure not to re-add, into the branch we merge into, any package
   that was removed from it, but still is in the branch we merge from:
   e.g. when merging `stable` into `devel`, it may be that
   `devel` had some packages removed (e.g. due to previously
   merging a topic branch into it, whose purpose is to *remove* custom
   packages). To this end, compare the resulting list of (package,
   version) in the `devel` APT suite with the one saved before
   the merge, check Git
   merges history if needed, apply common sense, and remove from
   `devel` the packages that were removed from it a while ago,
   and were just erroneously re-added by the merge operation.

<a id="workflow-reset"></a>

Resetting a suite to the state of another one
---------------------------------------------

   a. First, set some environment variables:

        # the suite to reset
        OLD=testing
        # the final state it should be in
        NEW=devel

   b. Then, empty the `OLD` suite:

      	ssh reprepro@incoming.deb.tails.boum.org \
      	      reprepro removematched $OLD '\*'

   c. Finally, merge `NEW` into `OLD`

      	ssh reprepro@incoming.deb.tails.boum.org \
      	      tails-merge-suite $NEW $OLD

<a id="workflow-merge-overlays"></a>

Merging APT overlays
--------------------

This operation merges all APT overlays listed in the given branch's
`config/APT_overlays.d/` into its own APT suite, empties
`config/APT_overlays.d/` accordingly, then commits and pushes to Git.

1. Set some environment variables:

        # The branch that should have its overlays merged
        BRANCH=devel

2. Merge the APT overlays in reprepro:

        git checkout "$BRANCH" && \
        for overlay in $(ls config/APT_overlays.d/) ; do
           if ! ssh reprepro@incoming.deb.tails.boum.org \
                   tails-merge-suite "$overlay" "$BRANCH" ; then
              echo "Failed to merge '$overlay' into '$BRANCH': $?" >&2
              break
           fi
        done

3. Empty `config/APT_overlays.d/`:

        git checkout "$BRANCH" && \
        git rm config/APT_overlays.d/* && \
        git commit config/APT_overlays.d/ \
           -m "Empty the list of APT overlays: they were merged"

4. Push the Git branch:

        git push origin "${BRANCH}:${BRANCH}"

<a id="workflow-post-tag"></a>

Tagging a new Tails release
---------------------------

Once the new release's Git tag is pushed, a cronjob creates
a new APT suite on the APT repository's side within a few minutes.
This new APT suite is called the same as the new release version.
One may check it has appeared in `~reprepro/conf/distributions`.

Then, the APT suite corresponding to the branch that was used to
prepare the release must be copied to the new empty APT suite that
just appeared:

	$ ssh reprepro@incoming.deb.tails.boum.org \
	     tails-merge-suite "$RELEASE_BRANCH" "$TAG"

<a id="workflow-post-release"></a>

After a new Tails release is out
--------------------------------

If you just put out a final release:

* [[merge `stable` or `testing` into
  `devel`|APT_repository#workflow-merge-main-branch]]
* increment the version number in devel's `debian/changelog` to match
  the next major release, so that
  next builds from the `devel` branch do not use the APT suite meant
  for the last release
* increment the version number in stable's `debian/changelog` to match
  the next point release, so that
  next builds from the `stable` branch do not use the APT suite meant
  for the last release

If you just released a RC:

* add a dummy changelog entry (for the upcoming, non-RC version) in
  the branch used for the release (`stable` or `testing`), so that the
  next builds from it do not use the APT suite meant for the RC
* add a dummy changelog entry (for the release *after* the one you
  released a RC for) in the branch used for the release (`stable` or
  `testing`), so that the next builds from it do not use the APT suite
  meant for the RC

If the release was a major one, then:

1. [[Hard reset the stable APT suite to
   the state of the testing one|APT_repository#workflow-reset]].

2. Empty `config/APT_overlays.d` in the `stable` branch:

        git checkout stable && \
        git rm config/APT_overlays.d/* && \
        git commit config/APT_overlays.d/ \
           -m "Empty the list of APT overlays: they were merged"

Giving access to a core developer
---------------------------------

1. Give SSH access to the `reprepro` user on the system that hosts
   reprepro (using the `ssh_authorized_key` Puppet resource).
2. Import the developer's public GnuPG key into the `reprepro` user's
   GnuPG keyring -- should be doable using Puppet, some day
3. Add the developer's OpenPGP key ID to `$reprepro_uploaders` in our
   `tails::reprepro` Puppet module. Deploy.

Contributing without privileged access
--------------------------------------

Non-core developers without access to the "private" APT infrastructure
would add the .deb they want to their Git branch as we have been
doing until now, push the result on GitLab or whatever... and at
merge time, we would rewrite their history to remove the .deb, and
import it into our APT repo.