[[!meta title="Automated testing tools"]]
We already have [[an automated test suite|contribute/release_process/test/automated_tests]].
This page is about tools that could allow us to improve it.
* Cucumber-like, in Python
* used by GNOME
* examples from the eog source tree, that use *behave* and *dogtail*:
- [steps definition](https://git.gnome.org/browse/eog/tree/tests/steps/steps.py)
* In [Debian](https://packages.qa.debian.org/b/behave.html) (in unstable since 2017/07/20)
* Python (with Jython) is now Sikuli's preferred scripting language;
it's also the language that has the best maintained bindings to
interact with libvirt, accessibility technologies, and more
* does *behave* work fine under Jython?
* actively maintained upstream as of 2017-05-31
* GUI test tool and automation framework written in Python
* uses Accessibility (a11y) technologies to communicate with
* used by GNOME in combination with *behave*: see the section about
* in Debian Wheezy, Jessie, and (as of 2016-02-14) Stretch;
but the package is orphaned
* how much do we still need Sikuli if we have dogtail?
* ships a `sniff` GUI that allows one to explore GUI applications
widgets and content using a11y technologies: very useful if we use
dogtail, or any similar tool
* most of the work is done by RedHat
## Misc helpers
* The pyatspi and Accessibility Python modules can be helpful
regardless of the actual tool we use. E.g. pyatspi can allow us to
listen for GUI events. Also see [[!debpkg gir1.2-atspi-2.0]] and
* [Accerciser](https://wiki.gnome.org/Apps/Accerciser), an interactive
accessibility explorer for the GNOME desktop
* "A tool for functional testing of GUI application."
* in Ubuntu, but not in Debian
* [usage documentation](https://developer.ubuntu.com/api/autopilot/python/1.5.0/)
LDTP is an open source testing tool that uses computer assistive
technology (accessibility) to automate GUIs. It is (was?) used by GNOME,
Mozilla and others:
* [[!wikipedia Linux_Desktop_Testing_Project]]
* latest upstream release (as of 2016-02-14) is 3.5, released mid-2013
* Debian Wheezy, Jessie and (as of 2016-02-14) Stretch have 2.3.1,
* The main bindings are Python, but there also are a Ruby client and
Perl bindings in the [Git repo](http://cgit.freedesktop.org/ldtp/ldtp2/tree/ldtp)
* The LDTP dev mailing list is very quiet, and it's unclear whether
GNOME still uses it, or instead switched to dogtail.
- Martin Pitt
umockdev ([source code](https://github.com/martinpitt/umockdev)),
a set of tools to record and mock hardware for debugging and testing
# Open questions
## Using accessibility technologies?
In some cases, it could simplify some testing steps, such as anything
about navigating menus, that we're currently mostly avoiding since
it's hard to do in a robust way with Sikuli.
A downside is that we're not exactly testing how most users interact
with the software. Some upsides are that it would ensure that our
stuff does support accessibility technologies, and that we would have
to maintain much less pictures.
## How to do it
We want to build a high-level Ruby library that helps us generate bits
of Python (that use e.g. dogtail) that we'll then send to the system
under testing (using the remote shell we have already), and run there.
For example, one feature this library should provide would be to
select a particular conversation tab in Pidgin, based on the
conversation tab's name; that is, using _text_ and not a picture; this
can be especially useful when we learn what name we are looking for
dynamically, at run time, and thus can't add the corresponding picture
Some stuff won't be doable in a nice, high-level way, so likely we
will need, for some corner cases, lower-level access to the generation
of the Python code that will be sent and run.