View Issue Details

IDProjectCategoryView StatusLast Update
0002349Slicer4Core: Extensionspublic2014-03-06 04:55
Reporterlassoan Assigned Tojcfr  
PriorityhighSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product Version 
Target VersionSlicer 4.2.0Fixed in VersionSlicer 4.2.0 
Summary0002349: No output is submitted to the dashboard if a module generic test fails
Description

It is hard to resolve module generic test failures, because the test does not give any clue about what went wrong: no output is printed on screen or submitted to the dashboard if a test fails.

It is on Windows, either Release or Debug mode.

How to reproduce:
-Introduce an error in the module (e.g., return with an empty list in QStringList qSlicerDicomRtImportModule::contributors()const)
-Build ExperimentalUpload
-Check the dashboard: the module test fails, but the test output is completely empty (see e.g., http://slicer.cdash.org/testDetails.php?test=2512032&build=29990)

TagsNo tags attached.

Activities

jcfr

jcfr

2012-07-26 02:53

administrator   ~0005247

It works on linux, I will check on windows.
See http://slicer.cdash.org/testDetails.php?test=2519068&build=30137

jcfr

jcfr

2012-07-26 03:14

administrator   ~0005249

Was able to reproduce it: http://slicer.cdash.org/viewTest.php?onlyfailed&buildid=30141

jcfr

jcfr

2012-08-22 13:38

administrator   ~0005737

Chris> Try to reproduce on windows, look at the XML files. Do no hesitate to talk with Zach and CMake folks.

crmullin

crmullin

2012-08-24 08:16

developer   ~0005776

I've submitted the experimental from the dashboard, and plenty of the module generic tests are failing with the error:
[C:/D/E/Slicer-build-2/Slicer-build/bin/Debug/qSlicerCamerasModuleCxxTests.exe] exit abnormally - Report the problem

These are reported on the dashboard.

crmullin

crmullin

2012-08-31 08:57

developer   ~0005906

Here is the XML submitted for the latest nightly build. It appears that the XML is fine, and the output simply isn't making it to the XML.

<Test Status="failed">
<Name>py_nomainwindow_qSlicerDICOMModuleGenericTest</Name>
<Path>./Modules/Scripted/Scripts/Testing/Python</Path>
<FullName>./Modules/Scripted/Scripts/Testing/Python/py_nomainwindow_qSlicerDICOMModuleGenericTest</FullName>
<FullCommandLine>C:\D\N\Slicer-1-build\Slicer-build\Slicer.exe "--no-splash" "--testing" "--ignore-slicerrc" "--no-main-window" "--python-code" "import slicer.testing; slicer.testing.runUnitTest(['C:/D/N/Slicer-1-build/Slicer-build/Modules/Scripted/Scripts/Testing/Python', 'C:/D/N/Slicer-1/Modules/Scripted/Scripts/Testing/Python'], 'qSlicerDICOMModuleGenericTest')"</FullCommandLine>
<Results>
<NamedMeasurement type="text/string" name="Exit Code"><Value>Timeout</Value></NamedMeasurement>
<NamedMeasurement type="text/string" name="Exit Value"><Value>1</Value></NamedMeasurement>
<NamedMeasurement type="numeric/double" name="Execution Time"><Value>900.048</Value></NamedMeasurement>
<NamedMeasurement type="text/string" name="Completion Status"><Value>Completed</Value></NamedMeasurement>
<NamedMeasurement type="text/string" name="Command Line"><Value>C:\D\N\Slicer-1-build\Slicer-build\Slicer.exe "--no-splash" "--testing" "--ignore-slicerrc" "--no-main-window" "--python-code" "import slicer.testing; slicer.testing.runUnitTest(['C:/D/N/Slicer-1-build/Slicer-build/Modules/Scripted/Scripts/Testing/Python', 'C:/D/N/Slicer-1/Modules/Scripted/Scripts/Testing/Python'], 'qSlicerDICOMModuleGenericTest')"</Value></NamedMeasurement>
<Measurement>
<Value/>
</Measurement>
</Results>
</Test>

jcfr

jcfr

2012-09-03 06:38

administrator   ~0005932

Good catch. Would also be nice to compare with the XML generated on linux. Then, you could probably try to contact CMake folks (kitware developer list) describing them the issue. Thanks

crmullin

crmullin

2012-09-05 14:00

developer   ~0005981

Here is the output of the same test on my linux machine:

http://slicer.cdash.org/testDetails.php?test=2872926&amp;build=38409

jcfr

jcfr

2012-09-05 14:14

administrator   ~0005982

What about the XML ? When you think you have enough element, make sure to engage CMake folks. Thanks

crmullin

crmullin

2012-09-07 10:54

developer   ~0006008

XML from my Linux machine:

<Test Status="failed">
<Name>py_nomainwindow_qSlicerDICOMModuleGenericTest</Name>
<Path>./Modules/Scripted/Scripts/Testing/Python</Path>
<FullName>./Modules/Scripted/Scripts/Testing/Python/py_nomainwindow_qSlicerDICOMModuleGenericTest</FullName>
<FullCommandLine>/home/crmullin/Projects/Slicer4-SuperBuild-Debug/Slicer-build/Slicer "--no-splash" "--testing" "--ignore-slicerrc" "--no-main-window" "--python-code" "import slicer.testing; slicer.testing.runUnitTest(['/home/crmullin/Projects/Slicer4-SuperBuild-Debug/Slicer-build/Modules/Scripted/Scripts/Testing/Python', '/home/crmullin/Slicer/Modules/Scripted/Scripts/Testing/Python'], 'qSlicerDICOMModuleGenericTest')"</FullCommandLine>
<Results>
<NamedMeasurement type="text/string" name="Exit Code"><Value>Failed</Value></NamedMeasurement>
<NamedMeasurement type="text/string" name="Exit Value"><Value>1</Value></NamedMeasurement>
<NamedMeasurement type="numeric/double" name="Execution Time"><Value>14.2729</Value></NamedMeasurement>
<NamedMeasurement type="text/string" name="Completion Status"><Value>Completed</Value></NamedMeasurement>
<NamedMeasurement type="text/string" name="Command Line"><Value>/home/crmullin/Projects/Slicer4-SuperBuild-Debug/Slicer-build/Slicer "--no-splash" "--testing" "--ignore-slicerrc" "--no-main-window" "--python-code" "import slicer.testing; slicer.testing.runUnitTest(['/home/crmullin/Projects/Slicer4-SuperBuild-Debug/Slicer-build/Modules/Scripted/Scripts/Testing/Python', '/home/crmullin/Slicer/Modules/Scripted/Scripts/Testing/Python'], 'qSlicerDICOMModuleGenericTest')"</Value></NamedMeasurement>
<Measurement>
<Value>Number of registered modules: 112
Register compression libraries
Number of instantiated modules: 112
Loading module "ACPCTransform"
Loading module "AddScalarVolumes"
Loading module "AffineRegistration"
Loading module "Annotations"
CreatePolyData: 0x30a2150
CreatePolyData: 0x30a43f0
CreatePolyData: 0x30a43f0
CreatePolyData: 0x30a43f0
CreatePolyData: 0x30a7390
CreatePolyData: 0x30a7390
CreatePolyData: 0x30a7390
Loading module "BRAINSDemonWarp"
Loading module "BRAINSFit"
Loading module "BRAINSFitEZ"
Loading module "BRAINSLandmarkInitializer"
Loading module "BRAINSROIAuto"
Loading module "BRAINSResample"
Loading module "BRAINSResize"
Loading module "BRAINSTransformConvert"
Loading module "BSplineDeformableRegistration"
Loading module "BSplineToDeformationField"
Loading module "BlobDetection"
Loading module "CLI4Test"
Loading module "Cameras"
Loading module "CastScalarVolume"
Loading module "ChangeTracker"
Loading module "CheckerBoardFilter"
Loading module "Colors"
Loading module "ConnectedComponent"
Loading module "CreateDICOMSeries"
Loading module "CropVolume"
Loading module "CurvatureAnisotropicDiffusion"
Loading module "DICOM"
Loading module "DICOM2FullBrainTractography"
Loading module "DICOMDiffusionVolumePlugin"
Loading module "DICOMScalarVolumePlugin"
Loading module "DICOMSlicerDataBundlePlugin"
Loading module "DTIexport"
Loading module "DTIimport"
Loading module "DWIJointRicianLMMSEFilter"
Loading module "DWIRicianLMMSEFilter"
Loading module "DWIToDTIEstimation"
Loading module "DWIUnbiasedNonLocalMeansFilter"
Loading module "Data"
Loading module "DataProbe"
Loading module "DicomToNrrdConverter"
Loading module "DiffusionTensorScalarMeasurements"
Loading module "DiffusionTensorTest"
Loading module "DiffusionWeightedVolumeMasking"
Loading module "DoubleArrays"
Loading module "EMSegment"
Loading module "EMSegmentCommandLine"
Loading module "EMSegmentQuick"
Loading module "EMSegmentTransformToNewFormat"
Loading module "Editor"
Loading module "Endoscopy"
Loading module "EventBroker"
Loading module "ExecutionModelTour"
Loading module "ExpertAutomatedRegistration"
Loading module "ExtractSkeleton"
Loading module "FiducialRegistration"
Loading module "GaussianBlurImageFilter"
Loading module "GradientAnisotropicDiffusion"
Loading module "GrayscaleFillHoleImageFilter"
Loading module "GrayscaleGrindPeakImageFilter"
Loading module "GrayscaleModelMaker"
Loading module "HistogramMatching"
Loading module "ImageLabelCombine"
Loading module "IntensityDifferenceMetric"
Loading module "LabelMapSmoothing"
Loading module "LabelStatistics"
Loading module "LinearRegistration"
Loading module "MaskScalarVolume"
Loading module "MedianImageFilter"
Loading module "MergeModels"
Loading module "ModelMaker"
Loading module "ModelToLabelMap"
Loading module "Models"
Loading module "MultiResolutionAffineRegistration"
Loading module "MultiVolumeExplorer"
Loading module "MultiVolumeImporter"
Loading module "MultiVolumeImporterPlugin"
Loading module "MultiplyScalarVolumes"
Loading module "N4ITKBiasFieldCorrection"
Loading module "OpenIGTLinkIF"
Loading module "OrientScalarVolume"
Loading module "OtsuThresholdImageFilter"
Loading module "OtsuThresholdSegmentation"
Loading module "PerformanceTests"
Loading module "ProbeVolumeWithModel"
Loading module "Reformat"
Loading module "ResampleDTIVolume"
Loading module "ResampleScalarVectorDWIVolume"
Loading module "ResampleScalarVolume"
Loading module "RigidRegistration"
Loading module "RobustStatisticsSegmenter"
Loading module "SampleData"
Loading module "SceneViews"
Loading module "SelfTests"
Loading module "SimpleRegionGrowingSegmentation"
Loading module "SlicerMRBTest"
Loading module "SubtractScalarVolumes"
Loading module "TestGridTransformRegistration"
Loading module "ThresholdScalarVolume"
Loading module "ThresholdThreadingTest"
Loading module "TractographyDisplay"
Loading module "TractographyFiducialSeeding"
Loading module "TractographyLabelMapSeeding"
Loading module "Transforms"
Loading module "VBRAINSDemonWarp"
Loading module "VectorToScalarVolume"
Loading module "ViewControllers"
Loading module "VolumeRendering"
Loading module "Volumes"
Loading module "VotingBinaryHoleFillingImageFilter"
Loading module "WebGLExport"
Loading module "Welcome"
Loading module "fiber_visibility_crash2438"
Loading module "labelToggleBug2049"
Number of loaded modules: 112
Register compression libraries
ctkDICOMModelPrivate::generateQuery: query is: SELECT UID as UID, PatientsName as Name, PatientsAge as Age, PatientsBirthDate as Date, PatientID as "Subject ID" FROM Patients ORDER BY "Name" DESC
ctkDICOMModelPrivate::updateQueries for Root: query is: SELECT UID as UID, PatientsName as Name, PatientsAge as Age, PatientsBirthDate as Date, PatientID as "Subject ID" FROM Patients ORDER BY "Name" DESC
QSqlQuery::exec: database not open
Register compression libraries
ctkDICOMModelPrivate::generateQuery: query is: SELECT UID as UID, PatientsName as Name, PatientsAge as Age, PatientsBirthDate as Date, PatientID as "Subject ID" FROM Patients ORDER BY "Name" DESC
ctkDICOMModelPrivate::updateQueries for Root: query is: SELECT UID as UID, PatientsName as Name, PatientsAge as Age, PatientsBirthDate as Date, PatientID as "Subject ID" FROM Patients ORDER BY "Name" DESC
PythonQt: QObject::connect() signal 'clicked()' does not exist on QAction
Uninstantiating: "labelToggleBug2049"
Uninstantiating: "fiber_visibility_crash2438"
Uninstantiating: "Welcome"
Uninstantiating: "WebGLExport"
Uninstantiating: "VotingBinaryHoleFillingImageFilter"
Uninstantiating: "VolumeRendering"
Uninstantiating: "ViewControllers"
Uninstantiating: "VectorToScalarVolume"
Uninstantiating: "VBRAINSDemonWarp"
Uninstantiating: "Transforms"
Uninstantiating: "TractographyLabelMapSeeding"
Uninstantiating: "TractographyFiducialSeeding"
Uninstantiating: "TractographyDisplay"
Uninstantiating: "ThresholdThreadingTest"
Uninstantiating: "ThresholdScalarVolume"
Uninstantiating: "TestGridTransformRegistration"
Uninstantiating: "SubtractScalarVolumes"
Uninstantiating: "SlicerMRBTest"
Uninstantiating: "SimpleRegionGrowingSegmentation"
Uninstantiating: "SelfTests"
Uninstantiating: "SceneViews"
Uninstantiating: "SampleData"
Uninstantiating: "RobustStatisticsSegmenter"
Uninstantiating: "RigidRegistration"
Uninstantiating: "ResampleScalarVolume"
Uninstantiating: "ResampleDTIVolume"
Uninstantiating: "Reformat"
Uninstantiating: "ProbeVolumeWithModel"
Uninstantiating: "PerformanceTests"
Uninstantiating: "OtsuThresholdSegmentation"
Uninstantiating: "OtsuThresholdImageFilter"
Uninstantiating: "OrientScalarVolume"
Uninstantiating: "OpenIGTLinkIF"
Uninstantiating: "N4ITKBiasFieldCorrection"
Uninstantiating: "MultiplyScalarVolumes"
Uninstantiating: "MultiVolumeImporterPlugin"
Uninstantiating: "MultiVolumeImporter"
Uninstantiating: "MultiVolumeExplorer"
Uninstantiating: "MultiResolutionAffineRegistration"
Uninstantiating: "Models"
Uninstantiating: "ModelToLabelMap"
Uninstantiating: "ModelMaker"
Uninstantiating: "MergeModels"
Uninstantiating: "MedianImageFilter"
Uninstantiating: "MaskScalarVolume"
Uninstantiating: "LinearRegistration"
Uninstantiating: "LabelStatistics"
Uninstantiating: "LabelMapSmoothing"
Uninstantiating: "IntensityDifferenceMetric"
Uninstantiating: "ImageLabelCombine"
Uninstantiating: "HistogramMatching"
Uninstantiating: "GrayscaleModelMaker"
Uninstantiating: "GrayscaleGrindPeakImageFilter"
Uninstantiating: "GrayscaleFillHoleImageFilter"
Uninstantiating: "GradientAnisotropicDiffusion"
Uninstantiating: "GaussianBlurImageFilter"
Uninstantiating: "FiducialRegistration"
Uninstantiating: "ExtractSkeleton"
Uninstantiating: "ExpertAutomatedRegistration"
Uninstantiating: "ExecutionModelTour"
Uninstantiating: "EventBroker"
Uninstantiating: "Endoscopy"
Uninstantiating: "Editor"
Uninstantiating: "EMSegmentTransformToNewFormat"
Uninstantiating: "EMSegmentQuick"
Uninstantiating: "EMSegmentCommandLine"
Uninstantiating: "EMSegment"
Uninstantiating: "DoubleArrays"
Uninstantiating: "DiffusionWeightedVolumeMasking"
Uninstantiating: "DiffusionTensorTest"
Uninstantiating: "DiffusionTensorScalarMeasurements"
Uninstantiating: "DicomToNrrdConverter"
Uninstantiating: "DataProbe"
Uninstantiating: "Data"
Uninstantiating: "DWIUnbiasedNonLocalMeansFilter"
Uninstantiating: "DWIToDTIEstimation"
Uninstantiating: "DWIRicianLMMSEFilter"
Uninstantiating: "DWIJointRicianLMMSEFilter"
Uninstantiating: "DTIimport"
Uninstantiating: "DTIexport"
Uninstantiating: "DICOMSlicerDataBundlePlugin"
Uninstantiating: "DICOMScalarVolumePlugin"
Uninstantiating: "DICOMDiffusionVolumePlugin"
Uninstantiating: "DICOM2FullBrainTractography"
Uninstantiating: "DICOM"
Uninstantiating: "CurvatureAnisotropicDiffusion"
Uninstantiating: "CropVolume"
Uninstantiating: "ResampleScalarVectorDWIVolume"
Uninstantiating: "Volumes"
Uninstantiating: "CreateDICOMSeries"
Uninstantiating: "ConnectedComponent"
Uninstantiating: "Colors"
Uninstantiating: "CheckerBoardFilter"
Uninstantiating: "ChangeTracker"
Uninstantiating: "CastScalarVolume"
Uninstantiating: "Cameras"
Uninstantiating: "CLI4Test"
Uninstantiating: "BlobDetection"
Uninstantiating: "BSplineToDeformationField"
Uninstantiating: "BSplineDeformableRegistration"
Uninstantiating: "BRAINSTransformConvert"
Uninstantiating: "BRAINSResize"
Uninstantiating: "BRAINSResample"
Uninstantiating: "BRAINSROIAuto"
Uninstantiating: "BRAINSLandmarkInitializer"
Uninstantiating: "BRAINSFitEZ"
Uninstantiating: "BRAINSFit"
Uninstantiating: "BRAINSDemonWarp"
Uninstantiating: "Annotations"
Uninstantiating: "AffineRegistration"
Uninstantiating: "AddScalarVolumes"
Uninstantiating: "ACPCTransform"

path: ['/home/crmullin/Projects/Slicer4-SuperBuild-Debug/Slicer-build/Modules/Scripted/Scripts/Testing/Python', '/home/crmullin/Slicer/Modules/Scripted/Scripts/Testing/Python']
testname: qSlicerDICOMModuleGenericTest

test_acknowledgementText (qSlicerDICOMModuleGenericTest.qSlicerDICOMModuleGenericTest) ... ok
test_application (qSlicerDICOMModuleGenericTest.qSlicerDICOMModuleGenericTest) ... ok
test_categories (qSlicerDICOMModuleGenericTest.qSlicerDICOMModuleGenericTest) ... ok
test_contributors (qSlicerDICOMModuleGenericTest.qSlicerDICOMModuleGenericTest) ... ok
test_logic (qSlicerDICOMModuleGenericTest.qSlicerDICOMModuleGenericTest) ... ok
test_module_valid (qSlicerDICOMModuleGenericTest.qSlicerDICOMModuleGenericTest) ... ok
test_setCurrentModule (qSlicerDICOMModuleGenericTest.qSlicerDICOMModuleGenericTest) ... ok
test_setMRMLScene (qSlicerDICOMModuleGenericTest.qSlicerDICOMModuleGenericTest) ... ok
test_settings (qSlicerDICOMModuleGenericTest.qSlicerDICOMModuleGenericTest) ... ok
test_widgetRepresentation (qSlicerDICOMModuleGenericTest.qSlicerDICOMModuleGenericTest) ... ok


Ran 10 tests in 0.167s

OK
QSqlDatabasePrivate::removeDatabase: connection 'SLICER' is still in use, all queries will cease to work.
QSqlDatabasePrivate::removeDatabase: connection 'DICOM-DB' is still in use, all queries will cease to work.
error: [/home/crmullin/Projects/Slicer4-SuperBuild-Debug/Slicer-build/bin/./SlicerApp-real] exit abnormally - Report the problem.
</Value>
</Measurement>
</Results>
</Test>

crmullin

crmullin

2012-09-10 11:08

developer   ~0006021

Here is the plain-text output from the last failed test (in the Temporary directory):

381/609 Testing: py_qSlicerDICOMModuleGenericTest
381/609 Test: py_qSlicerDICOMModuleGenericTest
Command: "C:/D/N/Slicer-1-build/Slicer-build/Slicer.exe" "--no-splash" "--testing" "--ignore-slicerrc" "--python-code" "import slicer.testing; slicer.testing.runUnitTest(['C:/D/N/Slicer-1-build/Slicer-build/Modules/Scripted/Scripts/Testing/Python', 'C:/D/N/Slicer-1/Modules/Scripted/Scripts/Testing/Python'], 'qSlicerDICOMModuleGenericTest')"
Directory: C:/D/N/Slicer-1-build/Slicer-build/Modules/Scripted/Scripts/Testing/Python
"py_qSlicerDICOMModuleGenericTest" start time: Sep 08 03:52 Eastern Daylight Time
Output:

<end of output>
Test time = 900.01 sec

Test Failed.
"py_qSlicerDICOMModuleGenericTest" end time: Sep 08 04:07 Eastern Daylight Time
"py_qSlicerDICOMModuleGenericTest" time elapsed: 00:15:00

After speaking with Zach Mullen: if the test is timing out, it won't submit any output unless it had some output beforehand.

crmullin

crmullin

2012-10-29 09:51

developer   ~0006856

Linux: OK
Mac OS X: OK
The problem occurs on Windows machines. It is not crucial for RSNA.

Note: When I look at this issue again, introduce an error manually as in the description.

jcfr

jcfr

2012-10-29 15:35

administrator   ~0006888

Fixed in r21276
See http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&amp;revision=21276

jcfr

jcfr

2014-03-06 04:53

administrator   ~0010748

Closing resolved issues that have not been updated in more than 3 months

Issue History

Date Modified Username Field Change
2012-07-25 10:38 lassoan New Issue
2012-07-25 10:38 lassoan Status new => assigned
2012-07-25 10:38 lassoan Assigned To => jcfr
2012-07-26 02:53 jcfr Note Added: 0005247
2012-07-26 02:53 jcfr Target Version => Slicer 4.2.0 - October 1st 2012
2012-07-26 03:14 jcfr Note Added: 0005249
2012-07-27 10:51 jcfr Assigned To jcfr => crmullin
2012-08-22 13:38 jcfr Note Added: 0005737
2012-08-24 08:16 crmullin Note Added: 0005776
2012-08-31 08:57 crmullin Note Added: 0005906
2012-09-03 06:38 jcfr Note Added: 0005932
2012-09-05 14:00 crmullin Note Added: 0005981
2012-09-05 14:14 jcfr Note Added: 0005982
2012-09-07 10:54 crmullin Note Added: 0006008
2012-09-10 11:08 crmullin Note Added: 0006021
2012-10-24 07:51 jcfr Priority normal => high
2012-10-29 09:51 crmullin Note Added: 0006856
2012-10-29 15:30 jcfr Assigned To crmullin => jcfr
2012-10-29 15:35 jcfr Note Added: 0006888
2012-10-29 15:35 jcfr Status assigned => resolved
2012-10-29 15:35 jcfr Fixed in Version => Slicer 4.2.0 - coming release
2012-10-29 15:35 jcfr Resolution open => fixed
2014-03-06 04:53 jcfr Note Added: 0010748
2014-03-06 04:55 jcfr Status resolved => closed