summaryrefslogtreecommitdiffstats
path: root/features/images
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'devel' into test/15460-replace-sikuli-force-all-testsintrigeri2020-03-217-0/+0
|\
| * Test suite: adjust UEFI test for GRUB (refs: #15806)intrigeri2020-02-213-0/+0
| |
| * Test suite: update reference images for overlayfs (refs: #17440)intrigeri2020-02-215-0/+0
| |
| * Merge remote-tracking branch 'origin/stable' into ↵intrigeri2020-01-142-0/+0
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | feature/8415-overlayfs+force-all-tests The stable branch got upgraded to the "single SquashFS diff" scheme (#15281), so we'll need the corresponding v2 UDFs and IUKs. As part of the conflict resolution, I'm therefore referencing them in the test suite, even though: - I did not create nor upload these IUKs yet. - I did not generate the corresponding UDFs and their signatures. My goal here is to have the test suite reflect what we should test nowadays, and fail in a more meaningful way.
| * | Test suite: update expected image (refs: #9373)intrigeri2019-12-062-0/+0
| | |
* | | Test suite: Dogtail-ify pin entry handling.anonym2020-02-041-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Our OpenCV-based image matching is so fast that it sometimes causes issues when we call maybe_deal_with_pinentry() twice in a row (e.g. for symmetric encryption): when clicking OK in the first pin entry we sometime manages to take the screenshot before the first pin entry prompt has disappeared, so we start doing the things intended for the second pin entry before it even has appeared, leading to failure.
* | | Test suite: bump some images.anonym2020-01-152-0/+0
| |/ |/| | | | | | | | | | | | | After bumping just these (objectively outdated) images I can run a large part of the test suite, indicating that OpenCV and Sikuli perform very similarly. Yay! Refs: #15460
* | Test suite: update reference image (refs: #15286)intrigeri2019-12-241-0/+0
| |
* | Test suite: add reference image (refs: #15286)intrigeri2019-12-231-0/+0
| |
* | Merge branch 'bugfix/17252-import-thunderbird-60' into ↵intrigeri2019-11-24218-0/+0
|\ \ | |/ | | | | feature/15281-single-squashfs-diff
| * Test suite: update image for Buster (refs: #14770)intrigeri2019-10-201-0/+0
| | | | | | | | | | | | Last time we updated it was for Jessie. I suspect it was already outdated for Stretch and might have been part of the reasons for #14770: the recovery_proc would never do anything useful.
| * Test suite: use sajolida's key instead of dkg'sintrigeri2019-10-201-0/+0
| | | | | | | | dkg's key is a victim of the certification flood attack :/
| * Merge branch 'feature/16693-tor-browser-9.0+force-all-tests' into testingintrigeri2019-10-201-0/+0
| |\ | | | | | | | | | Closes: #16693
| | * Revert "Test suite: update for the UI of Tor Browser 9's Tor Launcher" ↵intrigeri2019-10-201-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | (refs: #16693) Tor Launcher UI was fixed to allow entering multiple bridges again: https://trac.torproject.org/projects/tor/ticket/32154 This reverts commit 5f6ffd72d4aa39b85ea48f51a945c7c635b2698c.
| * | Test suite: make the "SSH is using the default SocksPort" scenario more ↵intrigeri2019-10-181-0/+0
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | robust (refs: #17163) This scenario failed 12 times in the last 16 test suite runs on the testing branch, so it's starting to be really annoying. Every time, SSH immediately fails with "nc: connection failed, SOCKSv5 error: General SOCKS server failure". I suspect there's a bug in our test suite, that makes us try to use Tor before it's ready. Regardless, we have a very similar test case in ssh.feature that is pretty robust: it never failed in the 16 test suite runs I've analyzed. I see two main potential reasons for this difference: - In tor_stream_isolation.feature we have the "I monitor the network connections of SSH" CPU hog running. This might make it harder for tor to do its job. - In ssh.feature, we use retry_tor so we retry up to MAX_NEW_TOR_CIRCUIT_RETRIES (default: 10); while in tor_stream_isolation.feature we only run SSH once. In short, the fragile scenario runs in a context that makes it more likely to fail, and it does not retry. So it's not very surprising that it's more fragile. Therefore, let's simply reuse the existing, robust implementation we have for a test connecting to a SSH server on the Internet. In passing, we get rid of one picture, which is always welcome.
| * Test suite: update Unsafe Browser proxy test for Tor Browser 9.0a8 (refs: ↵intrigeri2019-10-174-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | #17157) The UI this test used to rely on was removed from Tor Browser 9.0a8's code base, so let's re-implement it via prefs.js. And in passing, only test one Socksport (chosen randomly among those configured in torrc): this should be sufficient to verify that our firewall does not allow the clearnet user to connect to services listening on the loopback interface, which is what this scenario is essentially about.
| * Test suite: drop the "Unsafe Browser has no proxy configured" step (refs: ↵intrigeri2019-10-171-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | #17157) Our firewall blocks access from the clearnet user to the loopback interface, so proxies are not supposed to work. We validate this already with a combination of two tests: - "Scenario: The Unsafe Browser can load a web page" would fail if a proxy was configured and the firewall is set up as intended. - "Scenario: The Unsafe Browser cannot be configured to use Tor and other local proxies" would fail if the firewall was not set up correctly. So let's remove this extra test, which is hard to update since the UI it's currently using was removed from Tor Browser 9.0a8's code base.
| * Test suite: revert recent hacks for the "I open ..." step.anonym2019-10-091-0/+0
| | | | | | | | | | | | It turns out these hacks (introduced for refs: #17056) worked around a real bug, not a test suite issue, so we are gonna fix the real bug instead (refs: #17121).
| * Test suite: update Unsafe Browser reference image (refs: #17055)intrigeri2019-10-081-0/+0
| |
| * Test suite: update Unsafe Browser reference images (refs: #17055)intrigeri2019-10-082-0/+0
| |
| * Merge branch 'devel' into feature/16356-tor-browser-9.0+force-all-testssegfault2019-10-061-0/+0
| |\
| | * Test suite: re-add French-specific reference image for the Unsafe Browser ↵intrigeri2019-10-061-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | start page. This localized picture was previously removed by commit:396b0ead6d65ac42aa0374b076bf8ebbb24434f1. The string we're looking for is now translated into French, so we need to reintroduce a locale-specific reference image. Otherwise, when French is picked by supported_torbrowser_languages.sample(3), the "Unsafe Browser can be used in all languages supported in Tails" scenario will fail with "Unsafe Browser failed to launch in the following locale(s): fr_FR.utf8".
| * | Test suite: bypass Tor Browser GUI issues and open URLs via the command line ↵intrigeri2019-10-031-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (refs: #17056) On our slower isotesters, pasting the URL does not automatically open the Visit/Search menu, and the Enter key press is lost. I've also tried to use "Paste & Go" and to type the URL instead of pasting in; none of those helped. So let's have something that works, even if arguably it's not in the spirit of BDD. OTOH, it exercises opening URLs in the same way as they would if the user clicked on a link from another application, which is useful too.
| * | Test suite: update reference images for Tor Browser 9.intrigeri2019-10-035-0/+0
| | | | | | | | | | | | | | | | | | I've noticed, while trying to increase SIKULI_MIN_SIMILARITY, that these images were fuzzy. Let's try to avoid any issue caused by fuzzy matching.
| * | Test suite: update reference image for Tor Browser 9 (refs: #16356)intrigeri2019-10-031-0/+0
| | |
| * | Merge remote-tracking branch 'origin/devel' into ↵intrigeri2019-09-1212-0/+0
| |\ \ | | |/ | | | | | | feature/16356-tor-browser-9.0+force-all-tests
| | * Test suite: acknowledge that Evince and Tor Browser's print-to-file dialogs ↵intrigeri2019-09-072-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | are rendered in a subtly different manner. Since I've merged these 2 images in a1af7a5c8282e65be3b9972f76e07f4c2e0bd3fa and 9294b7d6ddef170eca6dc790f9475a98cadfb094, we've been doing something like this: 1. The Evince feature does not pass ⇒ let's update these images. 2. Now Tor Browser printing tests fail ⇒ let's update these images. 3. We see one of these images in sikuli_candidates, because "thanks" to #17029, the Evince tests pass anyway even though the image does not really match. But we think something along the lines of "this image is fuzzy, let's replace it with the Sikuli candidate". And then it's the turn of the Tor Browser tests to save their own Sikuli candidates. 4. Same as (3) but in the other direction. 5. I fix #17029. 6. I notice that the Evince tests don't pass anymore, so I update the images. 7. Now the Tor Browser tests don't pass anymore, while they did after (5). I dive into the problem and here we are. So yeah, I was wrong. There's a reason why we need 2 × 2 images here. Wrt. file naming, let's assume that Evince is the GTK 3 norm and Tor Browser deviates somehow. This does not matter much currently as no other tests use any of these images. It'll only matters if we ever start exercising printing from other GTK 3 apps.
| | * Test suite: update reference image.intrigeri2019-09-071-0/+0
| | | | | | | | | | | | | | | | | | We don't use fuzzy matches by default anymore, so some images need updating. Fuzzy matching previously hid the fact that this bookmark was actually missing (refs: #17030).
| | * Test suite: update reference image.intrigeri2019-09-061-0/+0
| | | | | | | | | | | | We don't use fuzzy matches by default anymore, so some images need updating.
| | * Test suite: update reference image.intrigeri2019-09-061-0/+0
| | | | | | | | | | | | We don't use fuzzy matches by default anymore, so some images need updating.
| | * Test suite: update reference image.intrigeri2019-09-061-0/+0
| | | | | | | | | | | | We don't use fuzzy matches by default anymore, so some images need updating.
| | * Test suite: Dogtail'ify all interactions with gedit (refs: #17028)intrigeri2019-09-066-0/+0
| | |
| * | Test suite: update reference image for Tor Browser 9 (refs: #17040)intrigeri2019-09-121-0/+0
| | |
| * | Test suite: update for the UI of Tor Browser 9's Tor Launcher (refs: #17038)intrigeri2019-09-121-0/+0
| |/ | | | | | | | | The UI now allows to enter only one bridge; pressing Enter in that text area triggers the "Connect" action.
| * Test suite: update 2 GTK file chooser dialog images.intrigeri2019-09-022-0/+0
| | | | | | | | | | | | | | The Buster version of these UI elements is quite different: both the icon and the text differ. Every test suite run results in updated images landing under sikuli_candidates. One wonders why the tests even pass with SIKULI_FUZZY_IMAGE_MATCHING set to false.
| * Test suite: update 2 gedit images.intrigeri2019-09-022-0/+0
| | | | | | | | | | | | | | | | | | The previous version of these images is fuzzy on Buster. Most of the time, Sikuli would retry, eventually find them on the screen, and save new versions in sikuli_candidates, which is why we did not notice this problem and update the reference images so far. But I've just seen one case when Sikuli could not recognize GeditPaste.png, while is was plain to see on the screen, so it's time to update these images.
| * Test suite: optimize performance of the "all notifications are disappeared" ↵intrigeri2019-09-011-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (refs: #17012) The robustness fix in the previous commit makes this step quite slow in the most common case — that is: a notification is displayed — because we'll have to wait for Dogtail to give up waiting on the "No Notifications" label, which is nowhere to be seen. Let's optimize for the case where there is at least a notification, such as the virtual machine warning, and look for the "Clear All" [notifications] button first. In passing, finish Dogtail'ifying this step by dropping its remaining Sikuli call.
| * Merge branch 'test/16281-misc+force-all-tests' into devel (refs: #16281)intrigeri2019-08-311-0/+0
| |\
| | * Test suite: make the "I set an administration password" step more robust.intrigeri2019-08-311-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On the devel branch, I've seen this failure mode: the first char of the admin password is lost because we typed it before the corresponding text field was displayed and focused. According to the debug log, we started typing the admin password 170 ms after clicking TailsGreeterAdminPassword.png, so it's not very surprising that occasionally we "win" this race against the Greeter. Let's at least wait for the "Administration Password" dialog to be visible on the screen before we interact with it.
| * | Test suite: update expected picture for the Unsafe Browser homepage (refs: ↵intrigeri2019-08-301-0/+0
| | | | | | | | | | | | | | | | | | #17004) This is the version for Buster. The version for Stretch may differ.
| * | Merge branch 'devel' into test/17004-unsafe-browser-homepage-busterintrigeri2019-08-30141-0/+0
| |\ \ | | |/
| | * Merge remote-tracking branch 'origin/devel' into ↵intrigeri2019-08-281-0/+0
| | |\ | | | | | | | | | | | | test/16004-adapt-usb-scenarios-to-usb-images+force-all-tests
| | | * Test suite: implement the PIM scenarios (refs: #15946)intrigeri2019-08-191-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I had to adjust the passphrase and secret file to match what's in the container with PIM segfault added: I can't easily generate one myself so let's use its settings everywhere. Also, refactor to avoid hard-coding things like "105 MB" in several places (and avoid adding more in this very commit).
| | * | Test suite: port more code to Dogtail that supports Python 3.intrigeri2019-08-191-0/+0
| | | |
| | * | Test suite: port some USB image code to Dogtail that supports Python 3.intrigeri2019-08-192-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I've seen a case when this click: @screen.wait_and_click('GnomeDisksStartRestoringButton.png', 10) … was lost. Dogtail should do better than this.
| | * | Test suite: test that we can install a USB image with GNOME Disks (refs: ↵intrigeri2019-08-192-0/+0
| | |/ | | | | | | | | | | | | | | | | | | | | | | | | #16004). This is about the documented upgrade process in which one downloads the new USB image, installs to an intermediary USB stick with GNOME Disks, and finally upgrades their main Tails USB stick from that intermediary USB stick with Tails Installer by cloning.
| | * Test suite: make image more specific (refs: #16820)intrigeri2019-06-191-0/+0
| | | | | | | | | | | | | | | Otherwise, for some reason Sikuli can't find it, as reported by anonym and confirmed by myself.
| | * Test suite: drop workaround to make the TAB spammer compatible with the UEFI ↵intrigeri2019-06-191-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | firmware (refs: #16820). As reported by anonym on #16820, and confirmed by my testing, pressing TAB doesn't seem to open the UEFI configuration, so the very reason why we had this workaround is gone.
| | * Revert "Update MemoryWipeCompleted.png" (refs: #16817)intrigeri2019-06-171-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I see lots of "FindFailed: can not find MemoryWipeCompleted.png" while the video shows that the expected message was displayed. And indeed, the candidates saved by Sikuli when it does manage to find that picture look more like what we have on our 3.x branches, which suggests that commit:56f58e34d2bca109dc2f7d2467d2bc1364732355 was perhaps only needed temporarily last month, perhaps due to #16743. This reverts commit 56f58e34d2bca109dc2f7d2467d2bc1364732355.
| | * Test suite: update fuzzy images for Buster (refs: #16634).intrigeri2019-06-174-0/+0
| | |