summaryrefslogtreecommitdiffstats
path: root/wiki/src/blueprint/freezable_APT_repository.mdwn
diff options
context:
space:
mode:
authorintrigeri <intrigeri@boum.org>2016-05-10 09:00:38 +0000
committerintrigeri <intrigeri@boum.org>2016-05-10 09:00:38 +0000
commit32f60ca65e5684f543e6e24878eb911d69ae93e2 (patch)
treea743f7049eae58b792e4e2cf23a3a86b336020e7 /wiki/src/blueprint/freezable_APT_repository.mdwn
parent8b6632df43199aae84efb2f1cf5d1a8e1ebcf290 (diff)
Update blueprint.
Diffstat (limited to 'wiki/src/blueprint/freezable_APT_repository.mdwn')
-rw-r--r--wiki/src/blueprint/freezable_APT_repository.mdwn57
1 files changed, 26 insertions, 31 deletions
diff --git a/wiki/src/blueprint/freezable_APT_repository.mdwn b/wiki/src/blueprint/freezable_APT_repository.mdwn
index 40127bc..f45d738 100644
--- a/wiki/src/blueprint/freezable_APT_repository.mdwn
+++ b/wiki/src/blueprint/freezable_APT_repository.mdwn
@@ -99,39 +99,34 @@ little value.
let's keep in mind that once generated, the config file should
be valid for a while.
e. **done** deny robots access to that data
- e. implement GC of expired snapshots and packages (`tails-delete-expired-apt-snapshots`):
- * **done** review [i]
- * **done** deploy with Puppet [i]
- * **done** test dry-run mode [i]
- * **done** test verbose mode [i]
- * **done** test silent mode [i]
- * **done** test removal of expired snapshots' `dist` directories after
+ e. **done** implement GC of expired snapshots and packages (`tails-delete-expired-apt-snapshots`):
+ * review [i]
+ * deploy with Puppet [i]
+ * test dry-run, verbose and silent modes [i]
+ * test removal of expired snapshots' `dist` directories after
`deleteunreferenced` [i]
+ * run via cron [i]
* evaluate performance [i]
- - very fast on all repositories except the `debian` one
- - `debian` repository: acceptable speed iff. `references.db`
- fits in the memory disk cache; so we'll need to monitor when
- it's not the case anymore, as that file is ever growing
- ([[!debbug 823629]]). The visible consequence would be:
- long periods of heavy disk read, and much slower snapshots
- expiration process; but we can also monitor the fill factor
- automatically with `db5.3_stat -d db/references.db -s
- references`, and the file size itself of course.
- When this problem occurs, we can either add more RAM
- to the VM if that's still feasible and reasonable, or reset
- the whole `debian` repository to an empty state (simple and
- big hammer, needs lots of coordination with developers), or
- to compact the DB, e.g. with `db5.3_dump` + `db5.3_load` or
- with a small program that runs `DB->compact()` in a loop
- with relevant options e.g. using Perl or Python bindings
- (<https://docs.oracle.com/cd/E17076_02/html/programmer_reference/am_misc_diskspace.html>,
- <https://groups.google.com/forum/#!topic/memcachedb/uiSvgStzYNY>,
- <https://docs.oracle.com/cd/E17076_02/html/api_reference/C/dbcompact.html>)
- * redirect output to a log file + logrotate snippet? or just run
- in silent mode and be done with it? [i]
- * clean up after failed experiments: the `debian` repo has tons
- of old, obsolete references that the script is not able to
- clean because the corresponding directories were removed already
+ f. handle ever-growing `references.db`, aka. [[!debbug 823629]]: if
+ `references.db` doesn't fit in the memory disk cache, then at
+ least our GC process gets very slow); the visible consequence
+ would be: long periods of heavy disk read, and much slower
+ snapshots expiration process; so we added an Icinga2 check on
+ the file size itself. When this problem occurs, our options are:
+ * add more RAM to the VM if that's still feasible and reasonable
+ (likely not)
+ * reset the whole `debian` repository to an empty state (simple
+ and big hammer, painful as it requires lots of coordination
+ with developers)
+ * compact the DB with `db5.3_dump` + `db5.3_load`: it got our
+ big `debian/db/references.db` down from 5.4 GB to 1.5 GB
+ * compact the DB with `DB->compact()`, using
+ [a script we wrote](https://git-tails.immerda.ch/puppet-tails/tree/files/reprepro/snapshots/time_based/tails-compact-reprepro-db)
+ (<https://docs.oracle.com/cd/E17076_02/html/programmer_reference/am_misc_diskspace.html>,
+ <https://groups.google.com/forum/#!topic/memcachedb/uiSvgStzYNY>,
+ <https://docs.oracle.com/cd/E17076_02/html/api_reference/C/dbcompact.html>):
+ works on small databases, but on our big `debian` the file
+ doesn't shrink
g. have build system output the snapshots being used,
and have Jenkins publish this info if available