summaryrefslogtreecommitdiffstats
path: root/wiki/src/blueprint/HTTP_mirror_pool.mdwn
blob: eecfec0b427150885204dd6ba2b02a48f0213cd4 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
**Ticket**: [[!tails_ticket 7161]]

[[!toc levels=3]]

# The plan

See the corresponding [[design document|contribute/design/mirrors]].

<a id="speed"></a>

# Speed

This is mainly for [[!tails_ticket 10295]]. We keep this information
around because it provides potentially useful data wrt. the reasons
why we picked some mirrors, but not others, for the fallback DNS pool.

## Fast & reliable enough mirrors

i.e. those that I've seen provide good speed and that have had no
reliability issue in the last N months.

Note: measurements done from lizard are capped to 100Mbps due to
upstream network configuration, so they can barely be used to reliably
compare those fast mirrors with each other. For measurements done from
Germany, upstream network should not be the limiting factor for most
practical purposes here.

* 5.45.108.219 aka https://tails.mirror.metalgamer.eu/tails/ (Germany):
  - one glitch in August, 2016
  - from lizard: 8.21 MB/s, 6.87 MB/s, 7.48 MB/s
  - from D.C.: 10.1 MB/s, 9.84 MB/s, 10.1 MB/s
  - from Germany: 37.7 MB/s, 43.4 MB/s, 37.2 MB/s
  - from France: avg. 21.1 MB/s, stdev 4.1 MB/s
  - from Netherlands: 50.4 MB/s, 41.3 MB/s, 43.7 MB/s
* 85.93.216.116 aka https://tails.c3l.lu/tails/ (Luxembourg):
  - from lizard: 6.58 MB/s, 6.72 MB/s, 3.73 MB/s, 5.52 MB/s, 2.97 MB/s, 5.31 MB/s, 4.46 MB/s, 4.50 MB/s, 3.15 MB/s
  - from D.C.: 8.76 MB/s, 8.82 MB/s, 9.51 MB/s
  - from Germany: 34.7 MB/s, 34.9 MB/s, 31.3 MB/s
  - from France: avg. 14.5 MB/s, stdev 3.4 MB/s
  - from Netherlands: 54.0 MB/s, 52.7 MB/s, 51.7 MB/s
* 195.154.14.189 aka https://16.dl.amnesia.boum.org/tails/ (France):
  - being moved to another IP (163.172.39.92 aka https://28.dl.amnesia.boum.org/tails/)
    so we're not adding it yet
  - from lizard: 5.08 MB/s, 5.25 MB/s, 6.26 MB/s, 6.33 MB/s, 6.17 MB/s
  - from D.C.: 4.65 MB/s, 7.21 MB/s, 7.01 MB/s
  - from Germany: 22.4 MB/s, 21.6 MB/s, 22.6 MB/s
  - from France: avg. 25.4 MB/s, stdev. 1.5 MB/s
  - from Netherlands: 17.2 MB/s, 17.5 MB/s, 18.4 MB/s
* 5.104.106.180 aka https://dl2.crypto-rebels.de/tails/ (Germany):
  - from lizard: 7.08 MB/s, 5.23 MB/s, 5.46 MB/s, 5.09 MB/s, 4.45 MB/s, 5.72 MB/s
  - from D.C.: 7.58 MB/s, 7.98 MB/s, 7.09 MB/s
  - from Germany (from the same network): 24.6 MB/s, 17.6 MB/s, 18.4 MB/s
  - from France: avg. 15.7 MB/s, stdev. 2.1 MB/s
  - from Netherlands: 38.0 MB/s, 37.9 MB/s
* 212.110.161.69 aka http://mirror.bytemark.co.uk/tails/ (UK):
  - from lizard: 5.31 MB/s, 6.62 MB/s, 4.61 MB/s, 6.70 MB/s, 6.34 MB/s, 6.26 MB/s
  - from D.C.: 7.65 MB/s, 6.68 MB/s, 7.57 MB/s
  - from Germany: 15.5 MB/s, 17.1 MB/s, 16.1 MB/s
  - from France: avg. 10.4 MB/s, stdev. 2.7 MB/s
  - from Netherlands: 25.2 MB/s, 66.3 MB/s, 43.9 MB/s
- 198.145.20.143 and 149.20.37.36, aka mirrors.kernel.org
- 208.80.154.15 aka mirrors.wikimedia.org
* 169.229.226.30 aka https://mirrors.ocf.berkeley.edu/tails/ (California)
  - from lizard: 10.3MB/s, 11.0MB/s, 11.2MB/s
  - from D.C.: 8.23 MB/s, 7.45 MB/s, 8.50 MB/s
  - from Germany: 5.01 MB/s, 6.25 MB/s, 5.90 MB/s
  - from Netherlands: 16.0 MB/s, 12.7 MB/s, 14.7 MB/s

## Too slow mirrors

* 62.201.161.88 aka http://tails.mirror.iphh.net/tails/ (Germany):
  - from lizard: 2.67 MB/s, 1.84MB/s, 1.82MB/s, 2.44MB/s
  - from Germany: 56.7MB/s, 32.1MB/s, 13.7MB/s
  - from France: avg. 11.6 MB/s, stdev 3.7 MB/s
* 178.217.184.32 aka https://tails.uk.to/tails/ (Poland):
  - from lizard: 4.96 MB/s, 4.96 MB/s
  - from Germany: 17.1 MB/s, 18.9 MB/s, 16.7 MB/s
  - from France: avg. 8.5 MB/s, stdev 2.1 MB/s
* 176.9.38.37:
  - from lizard: 2.81 MB/s, 2.74MB/s, 3.05MB/s, 2.74MB/s
  - from Germany: 43.0MB/s, 22.1MB/s, 7.41MB/s
  - from France: avg. 11.7 MB/s, stdev 2.6 MB/s
* 195.154.188.146: 3.69 MB/s
* 83.212.104.246:
  - from lizard: 3.90 MB/s
  - from France: avg. 4.8 MB/s, stdev 2.4 MB/s
* 188.138.127.35 aka https://tails.bl0m.de/tails/: perf. varies too much
* 45.33.79.99
  - from France: avg. 4.3 MB/s, stdev 0.6 MB/s
* 80.241.222.98 aka http://dl3.crypto-rebels.de/tails/ (Germany):
  - from lizard: 9.18 MB/s, 3.90 MB/s, 6.62 MB/s
  - from Germany: 7.83 MB/s, 8.00 MB/s, 7.54 MB/s
* 213.136.84.245 aka https://dl1.crypto-rebels.de/tails/ (Germany):
  - from lizard: 7.10 MB/s, 8.46 MB/s
  - from Germany: 9.74 MB/s, 7.99 MB/s, 7.60 MB/s
  - from France: avg. 6.5 MB/s, stdev. 2.4 MB/s
* 81.7.10.29 aka https://tails.ybti.net/tails/ (Germany):
  - from lizard: 6.28 MB/s, 5.17 MB/s, 5.26 MB/s
  - from Germany: 7.92 MB/s, 6.04 MB/s, 6.92 MB/s
  - from France: avg. 5.2 MB/s, stdev 1.5 MB/s
* 96.126.119.95 aka https://tails.interpipe.net/tails/ (USA):
  - from lizard: 6.10 MB/s, 7.04 MB/s
  - from Germany: 4.99 MB/s, 4.62 MB/s, 4.59 MB/s
  - from France: avg. 3.5 MB/s, stdev 0.1 MB/s
* 5.135.66.221 aka http://24.dl.amnesia.boum.org/tails/ (France):
  - from lizard: 3.27MB/s, 2.77MB/s, 2.89MB/s
  - from Germany: 6.22MB/s, 6.93MB/s, 5.05MB/s
  - from France: avg. 10.2 MB/s, stdev 1.7 MB/s
* 151.80.190.129 (France):
  - from lizard: 2.69MB/s, 1.24MB/s, 1.27MB/s
  - from Germany: 2.40MB/s, 2.60MB/s, 4.42MB/s
  - from France: avg. 7.0 MB/s, stdev 0.6 MB/s
* 158.36.190.173 (Norway):
  - from lizard: 3.17MB/s, 3.44MB/s, 2.44MB/s
  - from Germany: 24.4MB/s, 23.1MB/s, 23.5MB/s
  - from France: avg. 7.0 MB/s, stdev 1.1 MB/s
* 192.42.116.116 aka http://192.42.116.116/tails/ (Netherlands):
  - from lizard: 4.36 MB/s, 6.45 MB/s, 5.94 MB/s, 6.53 MB/s
  - from D.C.: 3.72 MB/s, 2.80 MB/s, 2.86 MB/s
  - from Germany: 45.0 MB/s, 45.5 MB/s, 38.0 MB/s
  - from France: avg. 16.8 MB/s, stdev 2.0 MB/s
  - from Netherlands: 89 MB/s, 94.8 MB/s, 88.0 MB/s
* 141.138.141.28 aka http://25.dl.amnesia.boum.org/tails/ (Netherlands):
  - from lizard: 3.35MB/s, 9.07MB/s, 6.00MB/s, 5.35 MB/s, 4.74 MB/s, 3.97 MB/s
  - from D.C.: 5.82 MB/s, 6.37 MB/s, 7.13 MB/s
  - from Germany: 16.7MB/s, 27.9MB/s, 24.5MB/s
  - from France: avg. 11.9 MB/s, stdev 2.3 MB/s
  - from Netherlands: 21.5 MB/s, 21.9 MB/s, 23.3 MB/s

## Not reliable enough mirrors

i.e. mirrors that have had issues at least once in the last 6 months;
let's not include them in the fallback DNS pool:

* 5.196.175.179
* 77.70.69.9
  - from France: avg. 0.3 MB/s, stdev 0.1 MB/s ⇒ **TODO** remove from the pool?
* 80.90.43.162
* 84.106.196.237
* 86.59.119.84
* 109.239.48.152
* 137.226.34.46:
  - from lizard: 798 KB/s, 9.80MB/s, 2.08MB/s, 1.37MB/s
     connection closes in the middle of the download pretty often
  - from Germany: 8.73MB/s, 10.4MB/s, 12.2MB/s
  - from France: avg. 11.1 MB/s, stdev 2.5 MB/s
* 141.138.136.78
* 144.76.14.145
* 149.202.98.175
* 178.32.220.171
* 192.99.131.144
* 198.199.103.96
* 212.47.229.219

# Initial research

See [[HTTP_mirror_pool/archive]].

<a id="HTTPS"></a>

# HTTPS mirrors

We've already switched all our mirrors in the Javascript mirror-pool, handled
by mirror-pool-dispatcher to HTTPS, but not all of our fallback mirrors
([[!tails_ticket 12833]]).

## Current problem space

Round-Robin pool

* we point to different IPs
* round robin incompatible with different CNAMES
* round robin uses IPs → incompatible with SSL certs
* Asking mirror OPs to create SSL certs themselves and keep them updated is not
  practicable.
* Links to dl.a.b.o on website & UDFs point to the round robin. (used for
  example on https://tails.boum.org/install/expert/usb/index.en.html)

* Website, DAVE2 and IUKs use Javascript based mirror-pool-dispatcher.
* Hardcoded URLs on the website need to be accessible & HTTPSified without
  Javascript

## Possible solutions

### Server based solution

We ruled this solution out when we first based the mirror-pool-dispatcher on
Javascript. Likely, we'd want to avoir recreating such a complicated solution
even if we will have to host our website ourselves and have this technical
possibility.

### One-mirror-only solution

A very stable and big mirror should become the only fallback for non-JS users
and the expert/wget installion method.

* → We ditch the round-robin
* → We monitor this server more often so that we can change it if ever it becomes inaccessible.

## Todo now

* deploy in lockstep on our live website:
  - change fallback_download_url_prefix in mirror-pool-dispatcher [u]
  - change all instances of http://dl.a.b.o → https://mirrors.wikimedia on our website [u]
     - except in UDFs
* ensure Tails 3.7 gets the updated mirror-pool-dispatcher submodule [i]
* ensure Tails 3.7 gets an updated `tails-perl5lib` package (`lib/Tails/MirrorPool.pm`) [i]
* prepare a branch in iuk.git that updates UDF generation code (replace dl.a.b.o with mirrors.wikimedia) [i]
* keep the fallback DNS pool running: it's still used by Tails Upgrader and we "support" skipping an upgrade (from 3.6 to 3.7) so it must remain working until 3.6 users can upgrade directly to 3.8
* prepare a branch against mirrors.git to document the new setup and drop the obsolete crap
* prepare a branch against tails.git to update the design doc

## Whenever we want

* tell wikimedia.org admins about our plans (before or after the change, whatever) [u]
* update the documentation for mirror operators in a dedicated Git branch: delete the part about dl.a.b.o [u]
* prepare a branch against mirror-pool.git that drops support for the DNS fallback pool [i]
* prepare a branch against puppet-tails.git that drops support for the DNS fallback pool [i]

## When releasing Tails 3.7 [bertagaz]

* all UDFs for upgrades must still have dl.a.b.o because Tails 3.6 and older
only support that (nothing special to do for that, just follow the release
process doc)

## When releasing Tails 3.8 [i]

* all UDFs for upgrades from 3.6 must have dl.a.b.o
* all UDFs for upgrades from 3.7 must have mirrors.wikimedia

## A few weeks after Tails 3.8 is released

* drop the dl.a.b.o fallback pool
* merge the branch into iuk.git
* merge the branch into mirror-pool.git
* merge the doc branch into mirrors.git
* merge the doc branch for mirror operators into tails.git
* merge the updated design doc branch into tails.git
* merge the branch into puppet-tails.git