Feb 2012:

Hi Folks,

The experiment I conducted wasn't successful. No errors or warnings were reported. See here.

What I did:
1) Made sure CTEST_USE_LAUNCHER is disabled at superbuild level before including CTest. But enabled in the subprojects.
2) Decoupled Launcher code from CTest/build_testing option. See https://gist.github.com/1840169

That said, I will create a small test case allowing us to easily work on the issue.

Thanks

I created a small toy project to investigate if the strategy proposed by CMake folks would work.

// ------
Project: https://github.com/jcfr/2468-TestCTestErrorReportingWithSuperbuild

The project is composed of:

• Two "external project" A and B that will create to static libraries
• A inner "external project" named Foo that will be built against A and B

Two build errors have been voluntarily introduced:

• Missing ";" at the end of line 17 in Foo.cxx
• Missing ";" at the end of line 9 in A.cxx

// ------
Results: http://slicer.cdash.org/index.php?project=Slicer4&date=2012-09-27#Experimental

// ------
The following experiments have been conducted:

1) WithoutCTestUseLauncher
=> Errors are reported with a chance of being interlaced

2) WithCTestUseLauncherAtTopLevel-UnpatchedCTest
=> No errors are reported

3) WithCTestUseLauncherInSubProject (PatchedCTest)

a) PatchedCTest
  => Errors are reported

b) PatchedCTest-TopLevelWarning
  => Errors are reported, top level warning happening at configure time is reported

c) PatchedCTest-TopLevelWarning-TopLevelBuildError
  => Toplevel error is NOT reported  !!!!!!!!!!!!!!!!!
  => SubProject errors are reported
  => Configure warning is reported

  => Running 'make' in the associated build tree gives:
         [ 25%] Built target A
         [ 50%] Built target B
         [ 75%] Built target C
         [100%] Built target Foo

     whereas build error are NOT reported !!!!!!!!!!

4) WithCTestUseLauncherAllLevel-PatchedCTest-TopLevelWarning-TopLevelBuildError

 => SubProject errors are reported
 => Top level configure warnings reported
 => Top level build error reported

// ------
Conclusion:

Code specific to Launcher existing in CTest.cmake should probably be moved into a file named "CTestLauncher.cmake". This should probably be included by default in every project independently of CTest

ExternalProject.cmake should be patched to automatically pass CTEST_USE_LAUNCHERS

Email sent to CMake folks:

Hi Folks,

I finally took the time to put together a small project and did some experiments. The details are reported here:

              http://www.na-mic.org/Bug/view.php?id=2468#c6258

// -------------
Results: http://slicer.cdash.org/index.php?project=Slicer4&date=2012-09-27#Experimental

Code: https://github.com/jcfr/2468-TestCTestErrorReportingWithSuperbuild/commits/master

// -------------
Based on the results, in order to have CTEST_USE_LAUNCHERS playing nicely with Superbuild so that all errors and warnings are reported, I suggest the following:

1) Code specific to Launcher existing in CTest.cmake should probably be moved into a file named "CTestLauncher.cmake". This file should probably be included by default in every project independently of CTest

2) ExternalProject.cmake should be patched to automatically pass CTEST_USE_LAUNCHERS

Knowing where would "CTestLauncher.cmake" could be included so that it's always considered, I could work on a patch.

Let me know what you all think.

Thanks
Jc

Hi Dave,

Here is a topic with the proposed change: https://github.com/jcfr/CMake/tree/fix-ctest-use-launchers-for-superbuild

I still think that enabling CTEST_USE_LAUNCHERS within a ctest script should allow the project and sub projects to report errors properly.

If instead of patching ExternalProject to pass the variable, there is an other mechanism allowing to achieve the same goal. I would be happy to help implementing it.

Currently, if the proposed topic, I still have to make sure every sub project is:

1) Including CTest:

Project like DCMTK are not including CTest at all, this project simply call "enable_testing()"

2) Properly passing CTEST_USE_LAUNCHERS in case of a superbuild setup used in the sub project itself.

example: Slicer is a superbuild depending on CTK which is also a superbuild.

Having a smarter mechanism to enable the launcher would be very very helpful for us.

Thanks for your help,

Hi Dave,

You'll have to change them anyway to include CTestUseLaunchers.... Why not change them to include(CTest)....?

Having a built-in mechanism allowing to enable the launcher would avoid to manually edit each project. The idea would be to automatically have "CTestUseLaunchers" included. Each time the project command is invoked for example.

Would such an approach be something possible ?

The only option that would make "no extra work required" here is to get the default value of it from an ENV var, and then set the ENV var in your outer script. That should be do-able.

Combined with the approach described earlier, enabling the launcher through ENV var would be great.

How should we proceed ?

Thanks

Topic pushed for CMake and CTK:
https://github.com/jcfr/CMake/tree/fix-ctest-use-launchers-for-superbuild
https://github.com/jcfr/CTK/compare/242-fix-superbuild-error-reporting

Currently testing locally

Fixed in r21112
See http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=21112

//--------------------
COMP: Fix CDash errors/warnings reporting for unix parallel build

Setting the variable CTEST_USE_LAUNCHERS_DEFAULT in the environment
to enable CTest launcher reliably will be supported by default in either
CMake 2.8.10 or 2.8.11.

In the mean time, your local installation of CMake can be modified based on
the following topic: https://github.com/jcfr/CMake/tree/fix-ctest-use-launchers-for-superbuild

Installation of CMake 2.8.8 have been updated on:

• macosx factory
• ubuntu factory
• dash21

See http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=21151

Hi Dave,

The patch has been updated and tested on the factory. See

https://github.com/jcfr/CMake/tree/fix-ctest-use-launchers-for-superbuild

http://slicer.cdash.org/viewBuildError.php?buildid=42565
http://slicer.cdash.org/viewBuildError.php?type=1&buildid=42565

To enable the launcher the following line have been added to the dashboard scripts: https://github.com/Slicer/Slicer/blob/master/CMake/SlicerDashboardDriverScript.cmake#L108-113

Let me know how you would like to proceed to get the patch integrated in the coming version of CMake.

Thanks for all your help,

Closing resolved issues that have not been updated in more than 3 months