Some email about the topic...

Oddly enough, although I did have the issue with the hanging (Slicer) and (CLIROITest) processes, after rebooting my machine I find that I cannot reproduce the issue either with my own local build or with the 2014-08-17 (latest) nightly.

Ron, I presume you are still seeing it today?

Jc, did you try with boggs?

-Steve

On Tue, Aug 19, 2014 at 11:03 AM, Steve Pieper <pieper@isomics.com> wrote:

Hi Jc - 

Don't forget you have access to boggs.bwh.harvard.edu - it's all yours to play with and it replicates the issue.  Feel free to login, run, reboot as much as needed.  It is a brand new mac pro with 10.9.4.

-Steve

On Tue, Aug 19, 2014 at 10:59 AM, Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com> wrote:

    Good news. Later this week, I will be able to borrow a laptop with MacOSX 10.9, that will be more handy to experiment.

    So far, here are the facts:

      Package  Slicer-4.3.1-2014-07-28-macosx-amd64.dmg  [1] : OK

      Package  Slicer-4.3.1-2014-08-05-macosx-amd64.dmg [2] : NOT OK

           => VTK6 / Clang 3.1 / Qt 4.8.6

    To narrow down the range, is there issue with the following ?

      Slicer-4.3.1-2014-08-02-macosx-amd64.dmg  [3]

             => VTK5 / g++-4.2.1 / Qt 4.8.5

    Thanks
    Jc

    [1] http://slicer.cdash.org/viewFiles.php?buildid=252367

    [2] http://slicer.cdash.org/viewFiles.php?buildid=255517

    [3] http://slicer.cdash.org/viewFiles.php?buildid=254343

    On Tue, Aug 19, 2014 at 10:47 AM, Steve Pieper <pieper@isomics.com> wrote:

        Interesting - and nothing else in parentheses then?  I'm debugging a different crash now but will try some more tests in a bit when I can reboot my machine.

        -Steve

        On Tue, Aug 19, 2014 at 10:41 AM, Ron Kikinis <kikinis@bwh.harvard.edu> wrote:

            BTW: the nightly build from 7/28 works fine. The problem must have been introduced between that date and 8/5 when I first reported it to you and JC

            On 8/19/14, 10:36 AM, Ron Kikinis wrote:

                Shows my level of incompetence.
                No CLIROITest. See attached

                On 8/19/14, 10:31 AM, Steve Pieper wrote:

                    Did you scroll through the full list Ron?  It may be on the second or
                    third page (use space bar to page).

                    On Tue, Aug 19, 2014 at 10:23 AM, Ron Kikinis <kikinis@bwh.harvard.edu
                    <mailto:kikinis@bwh.harvard.edu>> wrote:

                        see attached. I do not see either. I am using a bash term

                        On 8/19/14, 9:53 AM, Steve Pieper wrote:

                            Yes, makes sense.  I will keep looking.  Adding Jc to the cc
                    list.
                               Regarding the different build targets, I was planning to let
                            Jc know
                            that even my local build on 10.9.4 and the latest Xcode
                    exhibits the
                            same issue, so just retargeting the OS seems unlikely to be a
                    fix.

                            As a datapoint, the (Slicer) process on my machine has been
                    running
                            since 3pm on Saturday and my machine is okay.  With that zombie
                            in place
                            I can run slicer normally.

                            Also while looking I see there's also a process (CLIROITest)
                            that's been
                            running since the same time, so that must be the one hanging.

                            0000302 Slicer (master *$=)$ ps -elf | grep 3PM

                                501 89656   297     4006   0  31  0  2432784    708 -      S+
                                        0 ttys000    0:00.00 grep 3PM          9:51AM

                                501 86205   303     600e   0   0  0        0      0 -
                    ?E+
                                        0 ttys001    0:00.00 (Slicer)         Sat03PM

                                501 86227 86205     6006   0   0 20        0      0 -
                    ZN+
                                        0 ttys001    0:00.00 (CLIROITest)     Sat03PM

                            I don't see anything unusual about the source code for that, but
                            we can
                            investigate.

                            Ron, can you see if you also have one?  Do a "ps -elf | less"
                            and see if
                            you have any other processes in parentheses.  Then maybe delete
                            the one
                            that's hanging (either use the command line or right click and
                            pick Show
                            Package Contents) and try deleting it.

                            Inline image 1

                            I don't want to reboot my machine right now to test, but can do
                            so later
                            if needed.

                            -Steve

                            On Tue, Aug 19, 2014 at 8:23 AM, Ron Kikinis
                            <kikinis@bwh.harvard.edu <mailto:kikinis@bwh.harvard.edu>
                            <mailto:kikinis@bwh.harvard.__edu
                            <mailto:kikinis@bwh.harvard.edu>>> wrote:

                                 Hi Steve,

                                 thanks. The problem with the work around is that there
                            seems to be a
                                 memory leak. If I let the zombie go for too long, it
                    eventually
                                 freezes my machine.

                                 JC told me that he suspected that the problem would go
                            away, if he
                                 used a more up to date version of the compiler. I do not
                            know, if
                                 that is still on the table.

                                 Ron

                                 On 8/19/14, 8:06 AM, Steve Pieper wrote:

                                     Hi Ron -

                                     Are you referring to the issue with the hanging on
                            startup on
                                     mac?  I
                                     have looked into that a bit and I concur with Mike that
                            it's an
                                     OS level
                                     issue and it's possible that we won't be able to fix
                            it.  I have the
                                     issue on my computer, with either my own builds or
                    with the
                                     nightlies.

                                     It's a pain, but the workaround you identified works
                            for me: start
                                     slicer, let it hang, kill it, then start a new one.
                    It is
                                     confusing for
                                     users and it costs a core, so of course it's not the
                            long-term
                                     answer.

                                     Jc is also aware of the issue and we can discuss it
                            today on the
                                     developer hangout.  He can reproduce it and investigate
                            it on
                                     the new
                                     mac pro, but may also run out of options.

                                     -Steve

                                     On Tue, Aug 19, 2014 at 12:23 AM, Ron Kikinis
                                     <kikinis@bwh.harvard.edu
                            <mailto:kikinis@bwh.harvard.edu>
                            <mailto:kikinis@bwh.harvard.__edu
                    <mailto:kikinis@bwh.harvard.edu>>
                                     <mailto:kikinis@bwh.harvard.
                            <mailto:kikinis@bwh.harvard.>____edu

                                     <mailto:kikinis@bwh.harvard.__edu
                            <mailto:kikinis@bwh.harvard.edu>>>> wrote:

                                          Steve,
                                          Slicer does not run on Mac for me. Can you have a
                            look, please

                                          Ron

The hang went away for a while on my machine. But removing my settings caused it to come back. Now it happens all the time.

From Jc email - 2014-August-25:

Hi Folks,

I spent part of the weekend investigating and didn't resolve the issue.

Here are libraries I considered:

[QT-486_REGULAR] => Release 4.8.6 of Qt
[QT-486_PATCHED] => Release branch of Qt 4.8 and applied patches related to Qt issue 37699 and 40585.
[VTK510] => VTK 5.10 release
[VTK600] => VTK 6.0.0 release
[VTK610] => VTK 6.1.0 release
[VTK620-master-WITHKITS] => master branch of VTK with option VTK_ENABLE_KITS to reduce number of libraries
[PY273] => Usual Python 2.7.3
[PY278] => Upgraded CMake'ified python to version 2.7.8

... with the following setup:

[QT-486_REGULAR] + [PY273] + [VTK610] => FREEZE
[QT-486_PATCHED] + [PY273] + [VTK610] => FREEZE
[QT-486_PATCHED] + [PY278] + [VTK610] => FREEZE
[QT-486_PATCHED] + [PY278] + [VTK620-master-WITHKITS] => FREEZE
[QT-486_PATCHED] + [PY278] + [VTK510] => OK

In case of FREEZE:

Few settings I considered to try to start Slicer anyway:

   [DISABLEPYTHON]  => Starting with "--disable-python"
   [NOSTYLE] => Commented the code initiating Slicer style
   [NOMESSAGE_HANDLER] => Commented the code setting up the stdout capturing
   [NOLOADABLE_MOD_PYTHON] => Commented the code importing PythonQt + VTK wrapping for loadable modules
   [NOSCRIPTED_MOD] => Commented the code allowing scripted module to be loaded

   [DISABLEPYTHON] => OK

   [NOMESSAGE_HANDLER] => FREEZE

   [NOMESSAGE_HANDLER]+ [NOSTYLE] => FREEZE

   [NOMESSAGE_HANDLER]+ [NOSTYLE] + [NOLOADABLE_MOD_PYTHON] + [NOSCRIPTED_MOD]
                  => Slicer starts but FREEZE after interacting with 3D view for 1-2 secs
                  => Slicer starts but FREEZE after I open a dialog

Will try later this morning:

[NOMESSAGE_HANDLER]+ [NOSTYLE] + [NOLOADABLE_MOD_PYTHON] + [NOSCRIPTED_MOD] => Wait to check if it freeze

[QT-486_PATCHED] + [PY278] + [VTK610]  => Will see if commits between VTK 6.0.0 and VTK 6.1.0 introduced the problem.

Thanks,
Jc

Ps: See topic jcfr/Slicer@3809-python-v2.7.8-and-vtk for updated VTK and Python

From Steve - 2014-August-25:

Another data point:

I did a build against the system python [1] and got the same behavior - first run hangs, second runs while zombie is around.

But the zombie is holding onto the temp directory, so not everything in slicer works (CLIs won't run).

I was able to sample the zombie and also attach the debugger. The stack trace is always the same, in ImageLoader::weakBind called from a dlopen from python (see attached - Sample of slicer-syspy.txt )

-Steve

http://massmail.spl.harvard.edu/public-archives/slicer-users/2014/008334.html

From Steve - 2014-August-25:

A little more debugging:

I added an fprintf statement at the start of _PyImport_LoadDynamicModule in Slicer-superbuild/Python-2.7.3/Python/importdl.c.

fprintf(stderr, "_PyImport_LoadDynamicModule:\n\tname: %s\n\tpathname: %s\n\n", name, pathname);

With this I can see which import is hanging. Initially it was libvtkSlicerDoubleArraysModuleLogicPython.

But after removing that module and rebooting and running again it is now hanging in vtkSlicerMultiVolumeExplorerModuleLogicPython.

So I suspect that with vtk 6 we just have more dynamic libraries being loaded than the mac can handle.

According to the samples, the hang happens in strcmp in ImageLoader::weakBind, which is a tangled piece of work (search for weakBind in [1]) that might easily be buggy in extreme conditions within all the nested loops and conditional bookkeeping.

I noted that running slicer with --disable-loadable-modules does not prevent the hang from occurring (the vtkSlicer*Python shared libraries are imported even if the module is not activated).

If I move out all of lib/Slicer-4.3/qt-loadable-modules things get further, but then hang in python's own shared libraries (_heapq.so in this case).

Maybe we should be building VTK static, if such a thing is possible. Other thoughts?

-Steve

[1] http://www.opensource.apple.com/source/dyld/dyld-239.4/src/ImageLoader.cpp

With the change I implemented last night [1] and some more tweaks [2] on the Slicer side. I am now able to start and run Slicer without any freeze, load data, etc ... !

With these changes, there is still a crash when exiting Slicer on either Linux or MacOSX .. but that should not be too hard to address.

[1] https://github.com/Slicer/VTK/compare/Kitware:df24b795d8777f8eb02f6dd4470ed35252065ce1...slicer-v6.2.0-7167ad1_2014-08-21

[2] https://github.com/jcfr/Slicer/compare/Slicer:52c3ad8d99914f29eaad298954b59c40dad5043a...3809-python-v2.7.8-and-vtk

[3] https://github.com/jcfr/Slicer/tree/3809-python-v2.7.8-and-vtk

// ----------------------

Number of vtk*Python.so module:

VTK 5.10: 30 modules

VTK 6.0.0: 87 modules

With option VTK_ENABLE_KITS ON:

• VTK 6.2.0 (based of 7167ad1_2014-08-21): 90 modules

• VTK 6.2.0 (based of 7167ad1_2014-08-21 + improvements listed in [1]): 24

There's something more fundamental going on with the OS and Xcode and I don't think re-arranging the vtk kits is going to solve it.

I was just able to replicate the hang using the build below, which uses VTK 5 and system python, so none of the issues are specific to VTK 6 or our cmakified python.

rm -rf * ; cmake -DVTK_VERSION_MAJOR:STRING=5 -DSlicer_USE_SYSTEM_python:BOOL=ON -DSlicer_USE_SimpleITK:BOOL=OFF -DCMAKE_BUILD_TYPE:STRING=Release ../Slicer

Few things to consider:

• on Maverick and above, when configuring Slicer, I explicitly specify 10.8 as both SDK and deployment targets. If you don't specify 10.8, it will default to 10.9. And by default, the runtime used with 10.9 is libc++ .. which is incompatible with the runtime that was used to build Qt. See [1]

  In other word, make sure to specify these option when you configure Slicer:

  -DCMAKE_OSX_ARCHITECTURES=x86_64 \
  -DCMAKE_OSX_SYSROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk \
  -DCMAKE_OSX_DEPLOYMENT_TARGET=10.8

For example, with default being 10.9:

otool -L /path/to/VTKv6-build/lib/libvtkDICOMParser-6.2.dylib:
[...]
/usr/lib/libc++.1.dylib
[...]

If 10.8 being explicitly set:

otool -L  /path/to/VTKv6-build/lib/libvtkDICOMParser-6.2.dylib:
 [...]
 /usr/lib/stdc++.6.dylib
 [...]

Note that ideally, it should be sufficient to set CXXFLAGS with "-stdlib=libstdc++" but few things need to be fixed with flag propagation.

In the mean time, try again specifying 10.8.

• I also built against the release branch of Qt 4.8 built against SDK 10.8 (including on patches referenced above in the thread) ... I will check later if this is important or not.

Thanks
Jc

[1] http://stackoverflow.com/a/19637199/1539918

@Steve: This commit should avoid 10.9 from being selected by default. See https://github.com/jcfr/Slicer/commit/4c5045856de02b4fa9dab4e3ee4b6a697d899f08

Here is re-organized topic that only update VTK, that is what I will test and will most likely integrate:

https://github.com/jcfr/Slicer/compare/3809-update-vtk

Slicer r23619 to r23622 should help address part of the problem.

http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&amp;revision=23619
http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&amp;revision=23620
http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&amp;revision=23621
http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&amp;revision=23622

Marking as resolved. Let's see how tomorrow nightly package works out.

Alas it still happens with the 2014-08-26 version (today's nightly)

I can no longer replicate.

Per Jc the fix may be related to Quartz (2.7.7) and/or this OS patch [1] that is not offered by the standard Software Update process.

[1] http://support.apple.com/kb/DL1754

On my laptop, I lost the original X11 folders that contain the stub, if you still have them, could you try to remove the symbolic link, rename the folder and check how things go ?

The idea would be to revert the following:

sudo mv /usr/X11 /usr/X11.xstub
sudo ln -s /opt/X11 /usr/X11

sudo mv /usr/X11R6 /usr/X11R6.xstub
sudo ln -s /opt/X11 /usr/X11R6

I also notice that libtk8.6.dylib is not packaged.