View Issue Details

IDProjectCategoryView StatusLast Update
0002532Slicer4public2012-10-12 08:58
Reporterpieper Assigned Tosankhesh  
PriorityhighSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product Version 
Target VersionSlicer 4.2.0Fixed in VersionSlicer 4.2.0 
Summary0002532: old testing settings can impact new test results
Description

Because we always use the same testing configuration file ~/.config/www.na-mic.org/SlicerTesting.ini it can be corrupted from earlier tests and cause new tests to pass or fail incorrectly. We should probably start with a fresh testing settings file for every test.

Additional Information

On Thu, Sep 13, 2012 at 2:25 PM, Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com> wrote:

Hi Steve,

Thanks for looking into the issue :) See comment below.

On Wed, Sep 12, 2012 at 3:33 PM, Steve Pieper <pieper@ibility.net> wrote:

Yes - I agree some more output would be nice - I think the timeout is
handled by ctest, so I would have thought there would be some output
(think slicer just gets killed when it doesn't respond after a
while...)

In any case, I looked into things and it seems when we are in testing
mode we ignore the /user's/ settings, but use the same /testing/
settings so that the older version of the DICOM module, where it set
the directory by default, had set a valid directory on my mac test
machine, while the factory machine probably got a fresh test settings
file more recently. So the factory blocks on the file dialog, while
my test machine kept going.

I'm changing the behavior of the module so that if the testingEnabled
property of the application command options is set then the module
will use a temp directory. This will make the test (correctly) pass
on the factory.

Maybe we should always use fresh settings everytime the application is
started in --testing mode - that way we could avoid this kind of
cross-talk among tests...

That seems like a good option moving forward. Do you mind creating an issue
to capture that ?

Thanks
Jc

I'll check this in soon - I hope it fixes the factory issue, but I
agree with you Chris that the issue probably lies elsewhere.

-Steve

On Wed, Sep 12, 2012 at 2:58 PM, Christopher Mullins
<christopher.mullins@kitware.com> wrote:

Thanks Steve! This could very well not be what is hammering the machine
at
all. Maybe one of the tests that is GPU intensive would be a better
suspect.

It would be nice to have some output submitted in the case of a
'timeout,'
explaining that the database could not be found.

On Wed, Sep 12, 2012 at 2:04 PM, Steve Pieper <pieper@ibility.net>
wrote:

Hi Christopher -

The change in the link you sent (from 'open' to 'exec_') means that
the dialog is a blocking modal dialog that is prompting the user to
pick a dicom database directory if one has not been set already, which
is mean to prevent users from accidentally getting a database where
they don't expect it (or can't write to, like the application folder).

Yes, I can see that would lead to a timeout of the test if the
database directory does not exist for that user.

Oddly the test does not time out when I do 'make Experimental' on my
mac.

http://slicer.cdash.org/testDetails.php?test=2900724&amp;build=39118

I didn't write this test, but it looks like the test does not ignore
settings as it should - so it's using my database directory on my
machine, but blocking on the test machine where the database is not
set. When the database is not set, the 'timeout' is actually correct
behavior ;) But I don't know why it would make the machine crash.

I'll take a look at the test.

-Steve

On Wed, Sep 12, 2012 at 1:05 PM, Christopher Mullins
<christopher.mullins@kitware.com> wrote:

Hi all,

The short-term goal right now is to figure out which test is crashing
the
factory, and determine why. The prime candidates right now are the
tests
titled py_qSlicerDICOMModuleGenericTest and
py_nomainwindow_qSlicerDICOMModuleGenericTest because these appear to
be
the
tests that timeout instead of failing, and also submit no output.
Bill
Hoffman traced the failures back to the following commit:

http://viewvc.slicer.org/viewvc.cgi/Slicer4/trunk/Modules/Scripted/Scripts/DICOM.py?r1=20717&amp;r2=20718
Steve, since this is a DICOM commit, I defer to your judgement - is
this
something that would make a test timeout? The test appears to in
some
cases
open up a window and then run for 15 minutes. I've considered
disabling
those two tests to see if this helps, but let me know what you think.

We also determined that due to GPU usage, a long-term solution is
needed
to
facilitate restarting of both VM's and the host on a nightly basis.
This is
already captured by an issue in Mantis, and I have some ideas as to
how
this
could be implemented. Perhaps this solution would best wait until
after
the
release of 4.2, as this will require some restructuring of the way
the
tasks
are scheduled.

For the time being I'm taking care of the factory, so please assign
any
new
issues to me relating to the factory and I'll have a look. Thanks!

Chris

On Tue, Sep 11, 2012 at 1:43 PM, Jean-Christophe Fillion-Robin
<jchris.fillionr@kitware.com> wrote:

Would be awesome if you could provide ron and others with a quick
update
of what could be the issue.

Thanks
Jc

---------- Forwarded message ----------
From: "Julien Finet" <julien.finet@kitware.com>
Date: Sep 11, 2012 5:29 AM
Subject: Fwd: CDash - Slicer4
To: "Christopher Mullins" <christopher.mullins@kitware.com>
Cc: "Jean-Christophe Fillion-Robin" <JChris.FillionR@kitware.com>

---------- Forwarded message ----------
From: Ron Kikinis <kikinis@bwh.harvard.edu>
Date: Tue, Sep 11, 2012 at 7:37 AM
Subject: CDash - Slicer4
To: Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com>,
Julien
Finet <julien.finet@kitware.com>, Stephen Aylward
<Stephen.Aylward@kitware.com>, Steve Pieper <pieper@bwh.harvard.edu>

http://slicer.cdash.org/index.php?project=Slicer4

No mac nightly

--
Ron Kikinis, M.D.,
Robert Greenes Distinguished Director of Biomedical Informatics
Professor of Radiology, Harvard Medical School
Director, Surgical Planning Laboratory
http://www.spl.harvard.edu/~kikinis

--
+1 919 869 8849

TagsNo tags attached.

Activities

jcfr

jcfr

2012-09-24 07:25

administrator   ~0006183

The idea would be to for example ensure the Testing settings are deleted.
See https://github.com/Slicer/Slicer/blob/master/Base/QTCore/qSlicerCoreApplication.cxx#L755

sankhesh

sankhesh

2012-10-11 12:33

developer   ~0006500

Pushed fix here https://github.com/sankhesh/Slicer/tree/2532-Testing-settings

sankhesh

sankhesh

2012-10-12 08:03

developer   ~0006510

Fixed in r21164 ( http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&amp;revision=21164 )

pieper

pieper

2012-10-12 08:58

administrator   ~0006514

Looks like a good fix to me.

Issue History

Date Modified Username Field Change
2012-09-18 04:20 pieper New Issue
2012-09-18 05:54 jcfr Assigned To => sankhesh
2012-09-18 05:54 jcfr Status new => assigned
2012-09-18 05:54 jcfr Target Version => Slicer 4.2.0 - coming release
2012-09-18 05:54 jcfr Additional Information Updated
2012-09-24 07:25 jcfr Note Added: 0006183
2012-09-25 10:48 jcfr Priority normal => high
2012-09-25 10:48 jcfr Additional Information Updated
2012-10-11 12:33 sankhesh Note Added: 0006500
2012-10-12 08:03 sankhesh Note Added: 0006510
2012-10-12 08:03 sankhesh Status assigned => resolved
2012-10-12 08:03 sankhesh Fixed in Version => Slicer 4.2.0 - coming release
2012-10-12 08:03 sankhesh Resolution open => fixed
2012-10-12 08:58 pieper Note Added: 0006514
2012-10-12 08:58 pieper Status resolved => closed