|author||127.0.0.1 <127.0.0.1@web>||2019-10-06 02:12:28 +0000|
|committer||IkiWiki <ikiwiki.info>||2019-10-06 02:12:28 +0000|
This reverts commit 249ee8347e43e0b2cd32329cdfe26074f3a523c3
Diffstat (limited to 'wiki')
2 files changed, 138 insertions, 140 deletions
diff --git a/wiki/src/blueprint/GitLab.mdwn b/wiki/src/blueprint/GitLab.mdwn
index 37ad98b..3f01edf 100644
@@ -296,8 +296,6 @@ of forks, and follow the white GitHub rabbit(-hole):
from Trac to GitLab.
- They are considering using GitLab EE Ultimate, which includes
- - 2019-10-01 meeting: [agenda and notes](https://pad.riseup.net/p/e-q1GP43W4gsY_tYUNxf),
* KDE migration to Gitlab:
diff --git a/wiki/src/blueprint/translation_platform.mdwn b/wiki/src/blueprint/translation_platform.mdwn
index b5cfd20..6185a1e 100644
@@ -7,24 +7,24 @@ Git|contribute/how/translate/with_Git]]. This was a pretty high entry
barrier for new translators, especially those who are not familiar with
Git or the command line.
-This is the technical design documentation of our new setup.
-It is by no means perfect. We track known issues via
-[tickets on Redmine](https://redmine.tails.boum.org/code/projects/tails/issues?query_id=321).
+This is the technical design documentation of our setup.
-We also provide dedicated [[documentation on how translators can use
-translate our website.
+We also provide a dedicated [[documentation for translators on how to use
+Weblate|contribute/how/translate/with_Weblate]] to contribute
+translations. (This link will work once [[!tails_ticket 11763]] is
-Terminology used in this document
+Terms used in this document
-- Canonical Git repository: the [[main Tails Git
- repository|contribute/git#main-repo]] that our
- website is built from, in scripts often called "main repository" or "main
+- Canonical Git repository: the main Tails Git repository that our
+ website relies on, in scripts often called "main repository" or "main
- Production server: the server that hosts our website
- translate.lizard: the VM that hosts our Weblate web interface, the
- corresponding Git repositories, as well as the [staging website](https://staging.tails.boum.org/).
+ corresponding Git repositories, as well as the staging website.
+[Corresponding tickets on Redmine](https://redmine.tails.boum.org/code/projects/tails/issues?query_id=321)
Setup of our translation platform & integration with our infrastructure
@@ -32,29 +32,27 @@ Setup of our translation platform & integration with our infrastructure
We are using our own [Weblate instance](https://translate.tails.boum.org/).
Weblate uses a clone of the Tails main Git repository, to which
-translations get committed and pushed once they have been approved by a user with
+translations get committed, once they have been approved by a user with
reviewer status. Non-approved translations live on Weblate's database
-only, until they get reviewed. A [staging website](https://staging.tails.boum.org/) allows translators to
+only, until they get reviewed. A staging website allows translators to
preview non-reviewed translations in context.
Approved changes are automatically fed back into our canonical Git
-repository. This presents a major challenge, because we need to ensure
+repository. This presents a major challenge, indeed we need to ensure
-- No merge conflict occurs:
+- no merge conflicts occur:
- - such conflicts often occur in PO file headers which prevents Weblate
+ - such conflicts often occur on PO file headers which prevents Weblate
from automatically merging changes
- many contributors work on the same code base using different tools
(PO files can be edited by hand, using translation software such as
- Poedit, or they are generated by ikiwiki itself, which results in
+ POedit, or they are generated by ikiwiki itself, which results in
-- Only PO files are committed.
-- The committed PO files comply with shared formatting standards.
-- No compromised code is introduced.
+- only PO files are committed.
+- the committed PO files comply with shared formatting standards.
+- no compromised code is introduced.
In order to integrate Weblate and the work done by translators into our
process, we have set up this scheme:
@@ -64,13 +62,11 @@ process, we have set up this scheme:
Website and Weblate
-Our website uses ikiwiki and its [PO plugin](https://ikiwiki.info/plugins/po/).
-It uses markdown files for
+Our website uses ikiwiki and its PO plugin. It uses markdown files for
the English original language and carries a PO file for each translated
language. Thereby we distinguish languages that are activated on our
website from languages that have translations but are not yet activated
-on the website because they do not [[cover enough of
+on the website because they do not [[cover enough of a portion of
our core pages|contribute/how/translate/team/new/]] to be considered
@@ -86,18 +82,17 @@ appear in the Weblate interface. For example, the component:
-relates to the files `support.mdwn`, `support.es.po`, `support.de.po`, `support.pot`,
+relates to the files support.mdwn, support.es.po, support.de.po, support.pot,
-The repository used by Weblate is cloned and updated from the master
-branch of the Tails main
-repository. Changes generated on Weblate's copy
+The repository used by Weblate is cloned and updated from the Tails main
+repository, and its master branch. Changes generated on Weblate's copy
of the Tails main Git repository, located on the VM which hosts the
-Weblate platform, are automatically fed back to the master branch of
-the Tails main repository. This happens through a number of scripts,
+Weblate platform, are fed back to the Tails main repository, into the
+master branch, automatically. This happens through a number of scripts,
checks, and cronjobs that we'll describe below.
There are several languages enabled, some of them with few or no
@@ -111,14 +106,14 @@ or added as a remote:
git clone https://translate.tails.boum.org/git/tails/index/
-At the server the repository is located in:
+At the server the repository is located in
Weblate can commit to its local repository at any time, whenever
translations get approved. Changes done in the canonical repository by
Tails contributors via Git and changes done in Weblate thus need to be
-merged — in a safe place. This happens in an integration repository:
+merged - in a safe place. This happens in an integration repository:
@@ -130,37 +125,36 @@ website:
Automatic merging and pushing
-The integration of changes from the different repositories is done by a
-script which is executed on the VM hosting Weblate as [a cronjob](https://git-tails.immerda.ch/puppet-tails/tree/manifests/weblate.pp). The
-has the following steps which we will explain below:
+The integration work between the different repositories is done by a
+script which is executed on the VM hosting Weblate as [a cronjob every
+5 minutes](https://git-tails.immerda.ch/puppet-tails/tree/manifests/weblate.pp). The script
+ has the following steps which we will explain:
1. Canonical → Integration:
Update the integration repository with changes made on the
canonical repository (called "main" in the script).
- 2. Make Weblate locally commit any pending approved translation
+ 2. Make Weblate commit pending approved translations locally
3. Weblate → Integration:
Integrate committed changes from Weblate into the integration repository
4. Integration → Canonical:
- Push the up-to-date integration repository to the canonical repository.
+ Push up-to-date integration repository to canonical repository.
5. Canonical → Weblate:
- Pull from the canonical repository and update the Weblate components.
+ Pull from canonical and update the Weblate components.
6. Update Weblate's index for fulltext search
-Whenever a contributor modifies a markdown (`*.mdwn`) file and pushes
-to master, the corresponding PO files are updated, that is: the
+Whenever a contributor modifies a markdown (`*.mdwn`) file, and pushes
+to master, the corresponding POT and PO files are updated, that is: the
translatable English strings within those files are updated. This
- - on the production server itself, when [[building the
- - only for languages that are enabled on the production website.
+update happens on the production server itself, when [[building the
+wiki|contribute/build/website]] for languages that are enabled on the
We need to ensure on the translation platform server, that PO files for
additional languages (that are enabled on Weblate but not on the
-production website) are equally updated, committed locally, and pushed to
+production website) are equally updated, committed locally, pushed to
the canonical Git repository. On top of this we need to update Weblate's
-database accordingly, so that translatable strings can be updated for new or
+database accordingly, so that translations can be added for new or
modified English strings in those files, in all languages.
### Step 1: Canonical → Integration
@@ -173,54 +167,55 @@ merge changes into the (local) integration repository. The merge
strategy used for this step is defined in [`update_weblate_git.py`](https://git-tails.immerda.ch/puppet-tails/tree/files/weblate/scripts/update_weblate_git.py): (XXX: Shouldn't this script be called update_integration_git.py according to this documentation?)
When this script is executed, it merges changes in PO files based on
-single translation units (`msgids`). A merge conflict occurs when the same
-translation unit has been changed both in the canonical and the integration
+single translation units (`msgids`). Merge conflicts occur when the same
+translation unit has been changed in the canonical and the integration
repository (in the latter case, this would mean that the change has been
done via Weblate). In such a case, we always prefer the canonical
version. This makes sure that Tails developers can fix issues in
translations and have priority over Weblate.
-Due to this procedure we never end up with broken PO files. However, we
+Due to this procedure we never end up with broken PO files, however, we
may loose a translation done on Weblate.
Until here, only PO files of languages that are activated on our
-production website will be merged, as the production website
-does not refresh PO files for languages that are not activated there,
-so these PO files are outdated in the canonical Git repository at this point.
+production website will be merged, as the production website, i.e. the
+canonical Git repository does not regenerate PO files of non activated
Because of this limitation of ikiwiki, once the activated language PO
-files are merged, the script checks if PO files of other languages, that are not
-activated in production, need updating. We do this by
-generating POT files out of a PO file that we've previously defined as the
-default language. We do this for all components. If the actual POT
-file, generated on the production server, differs from the POT file we've
+files are merged, the script checks if PO files of additional,
+non-production activated languages need updating. We do this by
+generating POT files out of a PO file that we've previously defined as a
+default language. We do this for all componenets. If the actual POT
+file, generated on the production server differs from the POT file we've
just created, then every additional language PO file needs to be
On top of this, if the PO file of the default language (that is, its
-Markdown file) has been renamed, moved, or deleted, then the PO files of
-additional languages need to be accordingly renamed, moved, or deleted.
+markdown file) have been renamed, moved or deleted, than the PO files of
+additional languages need to follow this step.
In summary, our script applies all changes detected on the default
language to the additional languages.
The described mechanisms always `touch` files and change metadata, such
-as their `mtime`. That's why Git would always create a new commit for
+as `mtime`. That's why Git would normally always create a new commit for
such a change, but often those commits don't change the content of
files. In order to omit these empty unnecessary commits our script also
detects when a `fast-forward` is possible (the master branch is updated
-to HEAD of either the canonical or the integration branch). If only
+to HEAD of either the canonical or the integration branch). If only
Weblate or only modifications on the canonical repository introduces new
commits, a fast-forward can be done.
### Step 2: Trigger commits
-Weblate tries to minimize the number of commits (aka. "lazy
-commits"), so we need to explicitly to ask Weblate to commit every component
+Because Weblate tries to reduce the number of commits (aka. "lazy
+commits"), we need to ask Weblate explicitly to commit every component
which has outstanding changes since more than 24 hours.
This is done by triggering Weblate to commit pending approved
-translations using the internal command ([`manage.py commit_pending`](https://docs.weblate.org/en/weblate-2.20/admin/management.html#commit-pending)).
+translations using the internal command ([`manage.py
### Step 3: Weblate → Integration
@@ -233,9 +228,8 @@ integration repository. This is done by the script
Changes already present in the integration repository are preferred over
-the changes from the remote, Weblate repository. This makes fixes
-done to PO files manually, via the canonical Git repository, stick and propagate
+the changes from the remote, Weblate repository. This is to allow fixes
+done to PO files manually, via the canonical Git repository.
Again, PO file merges are done on translation units (`msgids`).
@@ -249,66 +243,63 @@ that exists in canonical.
After updating the Integration repository, we push the changes back to
-Canonical aka puppet-git.lizard. After this, the Canonical repository has
+Canonical aka puppet-git.lizard. After this the Canonical repository has
everything integrated from Weblate.
-On the side of the canonical Git repository, a Gitolite hook
-makes sure that Weblate only pushes changes on PO files.
+On the side of the canonical Git repository, gitolite has a special
+to make sure that Weblate is only allowed to push changes on PO files.
This hook also checks and verifies the committer of each commit, to make
sure only translations made on the Weblate platform are automatically
+pushed, and no other changes than those on PO files accepted. Otherwise
the push is rejected, for security reasons.
### Step 5: Canonical → Weblate
**Integrating the changes made in the Canonical Git repository into
-the Weblate repository**
After having merged changes from the canonical Git repository into the
integration Git repository, and integrated changes from Weblate there,
-we can assume that every PO file is now up-to-date, both in the Integration
-and Canonical repositories. Hence we can try to pull from the Canonical
-repository using a fast-forward only merge (`git pull --ff-only`). The
-canonical and Weblate repositories may see new commits anytime. This
-means: while our cronjob is running a new commit can be made. Then, a
-new commit on one side (canonical or Weblate), prevents a
-`fast-forward`. When this happens, the cronjob is run 5 minutes later
-anew, and then steps 1, 3 and 4 of the cronjob aim at fixing the cause of
-why the fast-forward was not possible this time.
-If the fast-forward merge was successful, we need to update Weblate's components
-to reflect the modifications that happened in Git, such as
+we can assume that every PO file now is up-to-date (in the Integration
+and Canonical repository). Hence we can try to pull from the Canonical
+repository using a fast-forward only (`git pull --ff-only`). Canonical and the
+Weblate repositories can get new commits everytime, also while the cronjob is
+running. It can happen, that a new commit on one side (Cannonical or Weblate)
+makes it impossible to perform a fast-forward. If we can't fast-forward the git
+repository, the cronjob is run 5min later anyways again, Than step 1,3 and 4 of
+the cronjob fixes the reason why the fast-forward was not possible this time.
+If the fast-forward was successful, we need to update Weblate's components
+to reflect the modifications that happened on the side of Git, such as
string and file updates, removals, renames, or additions. This is
handled by another script:
-Besides our scripts that modify the Weblate repository, Weblate itself
-keeps creating commits and updates the master branch. That's why the
-script is using a dedicated Git remote named `cron` to keep track of which
-commits need to be looked at for Weblate component changes. This remote
-name is set in
+We are not the only process touching the Weblate repository, as Weblate itself
+creating commits and updating the master branch That's why the script is using
+an own Git remote named `cron` to keep track of which commits need to look at
+for Weblate component changes. This remote name is set in the
-and used in the cronjob like this:
- update_weblate_components.py --remoteBranch=cron/master [...]
+and used in the cronjob `update_weblate_components.py
### Step 6
-Run [`manage.py update_index`](https://docs.weblate.org/en/weblate-2.20/admin/management.html#update-index).
-This updates Weblate's index for fulltext search.
-Weblate upstream authors recommend running it every 5 minutes.
+This updates Weblate's index for fulltext search. It is recommended by
+Weblate to run it every 5 mins.
In order to allow translators to see their non committed suggestions as
-well as languages which are not activated on <https://tails.boum.org>, we
-have put in place a [staging website](https://staging.tails.boum.org/).
+well as languages which are not activated on https://tails.boum.org we
+have put in place a [staging website](https://staging.tails.boum.org/) .
It is a clone of our production website and is regularly refreshed.
On top of what our production website has, it includes:
@@ -326,15 +317,19 @@ This allows:
- reviewers to check how translation suggestions look like on the
website, before validating them.
-### What is done behind the scenes to generate a new version of the staging website?
+ - check the sanity-check-website report:
+### What is done behind the scene to generate a new version of the staging website?
-cronjob is run.
This cronjob calls a script that extracts suggestions from Weblate's
database and applies them to a local clone of Weblate's Git repository,
-after having updated this clone:
+after having updated the clone with newer data from Weblate's VCS.
After that we run `ikiwiki --refresh` using an dedicated `ikiwiki.setup`
@@ -343,12 +338,6 @@ file for the staging website.
None of the changes on this repository clone are fed back anywhere and they
-### Sanity checks
-We automatically perform some sanity checks on this staging website.
-The last report of these checks is published on
Access control on the Weblate platform
@@ -390,8 +379,7 @@ Currently implemented proposal
- Every logged in user is in the `Users` group. Members of this group
are allowed to suggest translations but not to accept suggestions
- nor to directly save new translations of their own. They can also
- vote on suggestions.
+ nor to directly save new translations of their own.
- A reviewer, i.e. a member of the `@Review` group in Weblate, is
allowed to accept suggestions.
@@ -449,8 +437,10 @@ Currently implemented proposal
- - Is the resulting UX good enough? Does the ability to vote up
- suggestions helps sufficiently?
+ - Is the resulting UX good enough? Would it help if we allowed them
+ to vote up suggestions, even if this does not result in the
+ suggestion to be accepted as a validated translation?
+ (At the moment, suggestion voting is disabled.)
@@ -462,50 +452,60 @@ to the tmserver.
It is a very subtle way of increasing the quality of our translations.
-It should give suggestions when one is translating, under the translation
-window, in the _Machine translation_ tab.
+It should give suggestions when you are translating, under the translation
+window, tab 'Machine translation'.
-We use tmserver for machine translation ([upstream documentation](https://docs.weblate.org/en/weblate-2.20/admin/machine.html#tmserver)).
+We use tmserver for 'Machine translation'. You find the documentation
In order to update the suggestion we run
-[`update_tm.sh`](https://git-tails.immerda.ch/puppet-tails/tree/templates/weblate/update_tm.sh.erb) via cronjob every month.
+[`update_tm.sh`](templates/weblate/update_tm.sh.erb) via cronjob every month.
The tmserver can be queried like this [(see
+It should give suggestions when you are translating, under the translation
+window, tab 'Machine translation'.
-- [[Enabling a new language|contribute/l10n_tricks#weblate-administration]]<br/>
+- [[Enabling a new language|contribute/l10n_tricks#weblate-administration]]
Make sure to enable languages only if they are part of our tier-1
list or discuss the matter on the l10n-mailing list.
- Sysadmin: This documentation currently still lives in
- `translate-server.git` and might be moved somewhere else in the future
- ([[!tails_ticket 15086]]).
+ translate-server.git and should be moved somewhere else.
+ [[!tails_ticket 15086]]
Manually fix issues
-Our Weblate codebase is stored in `/usr/local/share/weblate`.
+We have our weblate codebase at
-If commands have to be run, they should be run as the `weblate` user;
-for example, with `sudo -u weblate COMMAND`.
+If commands have to be run, they should be run as user weblate (sudo -u weblate $COMMAND).
However, this VM is supposed to run smoothly without human
intervention, so be careful with what you do and please document
-modifications you make so that they can be fed back to a more
-appropriate place, such as our Puppet code or this document.
+modifications you make so that they can be fed back to puppet.git or other
+places if necessary.
-Reload translations from Git and cleanup orphaned checks and suggestions
+Reload translations from VCS and cleanup orphaned checks and suggestions
-If something went wrong, we may need to ask Weblate to reload all
-translations from Git, using the following command:
+The following commands mayby run manually if something went wrong:
+- Reload all translations from proper folder on disk (eg. in case you
+ did some updates in VCS repository) to components of weblate
+ `sudo -u weblate ./manage.py loadpo --all`
- sudo -u weblate ./manage.py loadpo --all
Weblate installation and maintenance
@@ -516,7 +516,7 @@ A hybrid approach
The Tails infrastructure uses Puppet to make it easier to enforce and
replicate system configuration, and usually relies on Debian packages to
ensure stability of the system. But the effort to maintain a stable
-system conflicts with installing and maintaining Weblate, a
+system somehow conflicts with installing and maintaining Weblate, a
Python web application, which requires using up-to-date versions of
Weblate itself and of its dependencies.