summaryrefslogtreecommitdiffstats
path: root/wiki/src/blueprint/automated_builds_and_tests/testing.mdwn
blob: cce2dc179d203d733137ce4f6089ee9d8a629a62 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
[[!meta title="Automated testing tools"]]

We already have [[an automated test suite|contribute/release_process/test/automated_tests]].
This page is about tools that could allow us to improve it.

[[!toc levels=2]]

# Tools

## behave

* [homepage](https://github.com/behave/behave)
* Cucumber-like, in Python
* used by GNOME
* examples from the eog source tree, that use *behave* and *dogtail*:
  - [feature](https://git.gnome.org/browse/eog/tree/tests/actions.feature)
  - [steps definition](https://git.gnome.org/browse/eog/tree/tests/steps/steps.py)
* In [Debian](https://packages.qa.debian.org/b/behave.html) (in unstable since 2017/07/20)
* Python (with Jython) is now Sikuli's preferred scripting language;
  it's also the language that has the best maintained bindings to
  interact with libvirt, accessibility technologies, and more
* does *behave* work fine under Jython?

## dogtail

* [homepage](https://gitlab.com/dogtail/dogtail)
* actively maintained upstream as of 2017-05-31
* GUI test tool and automation framework written in ​Python
* uses Accessibility (a11y) technologies to communicate with
  desktop applications
* used by GNOME in combination with *behave*: see the section about
  that one
* in Debian Wheezy, Jessie, and (as of 2016-02-14) Stretch;
  but the package is orphaned
* how much do we still need Sikuli if we have dogtail?
* ships a `sniff` GUI that allows one to explore GUI applications
  widgets and content using a11y technologies: very useful if we use
  dogtail, or any similar tool
* most of the work is done by RedHat

## Misc helpers

* The pyatspi and Accessibility Python modules can be helpful
  regardless of the actual tool we use. E.g. pyatspi can allow us to
  listen for GUI events. Also see [[!debpkg gir1.2-atspi-2.0]] and
  [[!debpkg gir1.2-wnck-3.0]].
* [Accerciser](https://wiki.gnome.org/Apps/Accerciser), an interactive
  accessibility explorer for the GNOME desktop

## autopilot

* [homepage](https://launchpad.net/autopilot)
* "A tool for functional testing of GUI application."
* in Ubuntu, but not in Debian
* [usage documentation](https://developer.ubuntu.com/api/autopilot/python/1.5.0/)

## LDTP

LDTP is an open source testing tool that uses computer assistive
technology (accessibility) to automate GUIs. It is (was?) used by GNOME,
Mozilla and others:

* [[!wikipedia Linux_Desktop_Testing_Project]]
* [homepage](http://ldtp.freedesktop.org/wiki/)
* latest upstream release (as of 2016-02-14) is 3.5, released mid-2013
* Debian Wheezy, Jessie and (as of 2016-02-14) Stretch have 2.3.1,
  released 2012-02-26
* [tutorial](http://download.freedesktop.org/ldtp/doc/ldtp-tutorial.pdf)
* The main bindings are Python, but there also are a Ruby client and
  Perl bindings in the [Git repo](http://cgit.freedesktop.org/ldtp/ldtp2/tree/ldtp)
* The LDTP dev mailing list is very quiet, and it's unclear whether
  GNOME still uses it, or instead switched to dogtail.

## misc

- Martin Pitt
  [announces](http://www.piware.de/2013/02/umockdev-record-and-mock-hardware-for-debugging-and-testing/)
  umockdev ([source code](https://github.com/martinpitt/umockdev)),
  a set of tools to record and mock hardware for debugging and testing

# Open questions

## Using accessibility technologies?

[[!tails_ticket 10721]]

In some cases, it could simplify some testing steps, such as anything
about navigating menus, that we're currently mostly avoiding since
it's hard to do in a robust way with Sikuli.

A downside is that we're not exactly testing how most users interact
with the software. Some upsides are that it would ensure that our
stuff does support accessibility technologies, and that we would have
to maintain much less pictures.

## How to do it

We want to build a high-level Ruby library that helps us generate bits
of Python (that use e.g. dogtail) that we'll then send to the system
under testing (using the remote shell we have already), and run there.

For example, one feature this library should provide would be to
select a particular conversation tab in Pidgin, based on the
conversation tab's name; that is, using _text_ and not a picture; this
can be especially useful when we learn what name we are looking for
dynamically, at run time, and thus can't add the corresponding picture
to Git.

Some stuff won't be doable in a nice, high-level way, so likely we
will need, for some corner cases, lower-level access to the generation
of the Python code that will be sent and run.