Review over gui test usage and architecture
Introduction
The current GUI tester implementation works but isn't easy to use. The provided functions are low-level and work at the Qt graphical component level. This results in redundancy, as a lot of boilerplate code gets duplicated. Furthermore, the tests are less readable.
The idea is to create higher level functions ("helper functions"), such as "push a button" or "load a volume" which may then be used in tests, to make both the writing and the reading easier.
The helper functions should be usable in external projects, and another helper functions could be created as well. The tests aren't executed in parallel inside a process, so concurrency isn't a problem. The helper functions should be straightforward to use. One should be able to easily make the link between the GUI test and the XML of the application being tested.
The helper functions should also provide a way to get a "backtrace", which would allow to understand in which situation exactly a test failed.
Architecture
Helper functions should be put in an architecture, which would most probably lie in a subfolder in libs/ui/testCore.
All static
One idea is to put these helper functions as static methods in classes. The problem with that solution is that static members are generally "infectious", and we end up with having static methods everywhere. The good point is that we don't have to deal with instantiation or inheritance.
Free functions
One idea is to not create methods and let these helpers functions as free functions. They would be put in their own files and their own namespace to have a good organization. Beside that, they have the same pros than All static. However, it might be less easy to transform them into classes, should the need for a data member arise.
The diagram should be the same as for All static, except that the helper classes are replaced by helper namespaces.
Hierarchy of helper classes separated from the hierarchy of the tester classes
One idea is to create an hierarchy of helper classes. An helper class may inherit from another one if it uses one of its function, for example if a "VideoTester" helper class needs to push buttons, it may inherit from "ButtonTester". The problem with that solution is that inheritance isn't really useful as these helper classes most likely won't have data.
Hierarchy of helper classes inherited by tester classes
Similar to the last idea, but the tester classes that want to use the helper function must inherit from the related helper class. For example, one both needs to test videos and buttons, he may inherit from "VideoTester" and "ButtonTester". The problem is that the tester classes may then inherit from a lot of helper classes, which may actually reduce readability for no advantages, and we may run into problem of diamond inheritance. The good point is that the inherited classes may have access to the same GUI Tester instance, which will remove the need to specify it for all helper function call.
How to get a backtrace
Exception trick
Use the stack unwinding mechanism of exceptions to rebuild the backtrace. The exception used should have an additional field that remember the backtrace. The problem is that there will be a lot of code that will be mostly reused for each created helper function.
void playVideo(Tester& tester, const std::string& playerId, const std::filesystem::path& videoFile){
try {
// ...
}catch(TesterAssertionFailed& e){
e.backtrace.push_back("playing video " + videoFile + " on player " + playerId);
throw;
}
// Handle other exceptions...
}
RAII trick
In Tester, a new field to remember a backtrace is added, as well as a method to add a new element to the backtrace, which also returns an RAII object. This RAII object will automatically remove the element on destruction, except if stack unwinding occurs because of an exception, because the backtrace will then be actually be useful.
void playVideo(Tester& tester, const std::string& playerId, const std::filesystem::path& videoFile){
auto bt = tester.addInBacktrace(tester, "playing video " + videoFile + " on player " + playerId);
// ...
}
Low level library calls (libbacktrace...)
We may use low level library that create backtraces for us, such as libbacktrace. However, as these solutions are low level, they are hardly portable. Moreover, it will display low level information, such as line number and exact name of called functions, which may not be useful for the final user.
Conclusion
For the architecture, I chose All static because inheritance seems overkill in this situation. All static is preferable to Free functions, as a class may become non-static, should the need of a non-static field arise.
For the backtrace generation, I chose the RAII trick as it results in less code being duplicated and is more modern.
With the new helper functions, ZoomOut test for SightViewer might be rewritten as:
namespace helper = sight::ui::testCore::helper;
helper::Button::push(tester, "toolBarView/Load series");
helper::SelectorDialog::select(tester, "VTK");
helper::FileDialog::fill(tester, utestData::Data::dir() / "sight/mesh/vtk/sphere.vtk");
helper::3DScene::zoom(tester, "genericSceneSrv", -10);
helper::Button::push(tester, "topToolbarView/Snapshot");
helper::FileDialog::fill(tester, "/tmp/LoadVtk.png");
compareImages(snapshotPath, referencePath);
Helper functions to create
Here is a list of helper functions that should be created. The list isn't exhaustive.
- Loading data
- Load video (path) (helper::VideoControls::load)
- Load volume (without clicking the button as it changes between application).
- Precise the reader type (helper::SelectorDialog)
- Choose the file path (helper::FileDialog)
- Dialogs
- Choose a path for file dialogs (helper::FileDialog)
- Select something from a selector dialogs (helper::SelectorDialog)
- Dismiss a message dialog (helper::MessageDialog)
- Component-level interaction
- Click on a button (helper::Button)
- Move a slider (helper::Slider)
- Fill a field (helper::Field)
- Check if a label has the good value (helper::Label)
- 3D view interaction (helper::3DScene)
- Zoom in/out (helper::3DScene::zoom)
- Move the camera (helper::3DScene::moveCamera)
- Video interaction (helper::VideoControls)
- Start the video (helper::VideoControls::start)
- Pause the video (helper::VideoControls::pause)
- More basic utility functions
- Check if a component is hidden (Tester::isHidden)
- Save activities (?)
- Take a screenshot of the whole application or of a component (Tester::takeScreenshot)
- Check if a component is present (Tester::isPresent)