I am currently working on a new class qSlicerCoreTestingUtility under Slicer/Base/QTCore that defines overloaded image compare methods. This class uses vtkTesting::RegressionTest to do the image compare.

The initial effort is to integrate image comparison with the current testing framework. Next, I will add a test driver and modify the existing cmake macro "slicer_add_python_test" that eliminates the need for the additional python file per xml script. The idea is that when someone records a test, all they need to do is add a line -
slicer_add_test(....)
in the CMakeLists.txt file. This streamlines the process and makes it easy for adding tests.
Next step would be move all the testing data (baseline images)( http://na-mic.org/Mantis/view.php?id=2060 ) onto Midas and have testing pull the baselines directly.

Pushed new topic with initial changes:
https://github.com/sankhesh/Slicer/tree/2567-screenshot-based-testing

Update: Added some versatility methods.
Am currently testing implementation. Came across an issue with ctkLayoutManager that does not allow access to viewport using python.( https://github.com/commontk/CTK/issues/248 )

There seems to be an issue with the image comparison implementation when I give it a QWidget and it has to grab a screenshot and compare with baseline image.

FYI - Commit associated with CTK issue 248 has been integrated to CTK.

Made some progress on the topic. Added a convenience cmake macro to add new qttesting tests and removed dependency on additional python script. Next, going to change existing qttesting based tests to use new macro as example.

Testing Data Baselines are now on Midas. Added functionality to fetch baselines while building for qttesting tests using ExternalData.cmake. Kindly review.

Copied from note added by mistake to issue 2060 on Dec 2012-12-07:

// -----------------------------------------
Jc: QtTesting baseline should be added in a baseline folder here: http://slicer.kitware.com/midas3/folder/832

Consider also reading: http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/Tutorials/UploadTestData

// --------------
From email to Sankhesh Dec 4:

Hi Sankhesh,

1. If you manually run the test using "ctest -R <name-of-test>", does it locally fails ?

2. This variable is for the upload of Slicer package. To download test data from midas, there are currently two mechanism :(

   a) one using ExternalData.cmake: Data are downloaded at build time. See https://github.com/Slicer/Slicer/blob/master/CMakeLists.txt#L536-543

   • usage example: https://github.com/Slicer/SkullStripper/blob/master/Testing/Cxx/CMakeLists.txt

   b) one using Midas.cmake: Data are downloaded at test time. See https://github.com/Slicer/Slicer/blob/master/CMakeLists.txt#L557-560

   • usage example: https://github.com/Slicer/Slicer/blob/master/Modules/CLI/DicomToNrrdConverter/ExtendedTesting/CMakeLists.txt

For the qttesting baseline, let's use the ExternalData approach.

Moving forward, the idea would be to consolidate ExternalData.cmake to also provide the functionality of Midas.cmake. Is is something you would like to work on ?

// -----------------------------------------
Sankhesh:

Added functionality to specify a URL for baseline image that will be downloaded before testing.
Where should the baseline images for qttesting tests be uploaded on Midas and is there a way to generate the Midas download URL if only the item id is provided?

For reference: https://github.com/sankhesh/Slicer/commits/2567-screenshot-based-testing [^]

Would be great if you could rebase the topic against the current Slicer master branch.

Latest code pushed based on suggestions. Kindly review.

Thanks. See comments: https://github.com/sankhesh/Slicer/commits/2567-screenshot-based-testing

Cleaned up branch and squashed minor commits. Kindly review and let me know your comments/suggestions.

https://github.com/sankhesh/Slicer/commits/2567-screenshot-based-testing