View Issue Details

IDProjectCategoryView StatusLast Update
0004394Slicer4Core: Building (CMake, Superbuild)public2017-08-03 09:07
Reporterpieper Assigned Tojcfr  
PrioritynormalSeveritymajorReproducibilityhave not tried
Status closedResolutionfixed 
PlatformlinuxOSdebianOS Version
Product VersionSlicer 4.7.0 
Target VersionSlicer 4.7.0Fixed in VersionSlicer 4.7.0 
Summary0004394: race condition installing python modules (e.g. gitdb)
Description

Building with make -j can lead to an error as pasted below. It looks like two easy_install processes are both trying to access a lock file at the same time.

Additional Information

Adding gitdb2 2.0.0 to easy-install.pth file

Installed /home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/gitdb2-2.0.0-py2.7.egg
Processing dependencies for gitdb2==2.0.0
Searching for smmap2==2.0.1
Best match: smmap2 2.0.1
Processing smmap2-2.0.1-py2.7.egg
smmap2 2.0.1 is already the active version in easy-install.pth

warning: no files found matching 'VERSION'
warning: no files found matching 'CHANGES'
warning: no files found matching 'README'
warning: no files found matching 'gitdb/_fun.c'
warning: no files found matching 'gitdb/_delta_apply.c'
warning: no files found matching 'gitdb/_delta_apply.h'
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '
.pyc' found anywhere in distribution
warning: no previously-included files matching '.so' found anywhere in distribution
warning: no previously-included files matching '
.dll' found anywhere in distribution
warning: no previously-included files matching '*.o' found anywhere in distribution
Traceback (most recent call last):
File "setup.py", line 43, in <module>
"Programming Language :: Python :: 3.5",
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/install.py", line 67, in run
self.do_egg_install()
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/install.py", line 117, in do_egg_install
cmd.run()
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/easy_install.py", line 411, in run
self.easy_install(spec, not self.no_deps)
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/easy_install.py", line 654, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/easy_install.py", line 701, in install_item
self.process_distribution(spec, dist, deps)
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/easy_install.py", line 756, in process_distribution
self.easy_install(dist.as_requirement())
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/easy_install.py", line 673, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/easy_install.py", line 704, in install_item
self.process_distribution(spec, dists[0], deps, "Using")
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/easy_install.py", line 721, in process_distribution
self.update_pth(dist)
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/easy_install.py", line 1182, in update_pth
self.pth_file.save()
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/easy_install.py", line 1595, in save
self.lockfile.release()
File "/home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/setuptools-35.0.1.post20170718-py2.7.egg/setuptools/command/lockfile/linklockfile.py", line 59, in release
raise NotMyLock("%s is locked, but not by me" % self.path)
setuptools.command.lockfile.NotMyLock: /home/researcher/Slicer-superbuild/python-install/lib/python2.7/site-packages/easy-install.pth is locked, but not by me

CMake Error at /home/researcher/Slicer/CMake/ExternalProjectForNonCMakeProject.cmake:76 (message):
python-gitdb: Error in install step. See
/home/researcher/Slicer-superbuild/python-gitdb_install_step_output.txt and
/home/researcher/Slicer-superbuild/python-gitdb_install_step_error.txt
Call Stack (most recent call first):
/home/researcher/Slicer-superbuild/python-gitdb_install_step.cmake:3 (ExternalProject_Execute)

CMakeFiles/python-gitdb.dir/build.make:73: recipe for target 'python-gitdb-prefix/src/python-gitdb-stamp/python-gitdb-install' failed
make[2]: [python-gitdb-prefix/src/python-gitdb-stamp/python-gitdb-install] Error 1
CMakeFiles/Makefile2:330: recipe for target 'CMakeFiles/python-gitdb.dir/all' failed
make[1]:
[CMakeFiles/python-gitdb.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

TagsNo tags attached.

Activities

pieper

pieper

2017-07-18 12:29

administrator   ~0014927

If I run without -j to get past the python build I am able to stop it and restart with -j and the build continues past the error spot.

lassoan

lassoan

2017-07-24 22:02

developer   ~0014941

Is this a documentation issue only (describe that -jX is fast but most likely you need to run multiple times) or there is a chance that this can be fixed?

jcfr

jcfr

2017-07-24 22:21

administrator   ~0014943

The original issue is that setuptools does not support parallel build. In Slicer we used a modified version that attempt to support it. It is based on an original patch found in sagemath.

See

https://github.com/Slicer/setuptools/commit/2bea6fc523f0cf58dacab21c723f105c3652ecff

https://github.com/Slicer/Slicer/commit/f0cb110f75d65bb1f8fcbd334787133239a5c616

https://github.com/Slicer/Slicer/commit/a3deed458f4f5d7219a96f828c63ee56832d68d3

jcfr

jcfr

2017-08-03 00:12

administrator   ~0015004

Fixed in r26191
See http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&amp;revision=26191 and https://github.com/Slicer/Slicer/pull/761

pieper

pieper

2017-08-03 09:07

administrator   ~0015005

Seems to be fixed - thanks!

Issue History

Date Modified Username Field Change
2017-07-18 11:47 pieper New Issue
2017-07-18 11:47 pieper Status new => assigned
2017-07-18 11:47 pieper Assigned To => jcfr
2017-07-18 12:29 pieper Note Added: 0014927
2017-07-24 22:02 lassoan Note Added: 0014941
2017-07-24 22:03 lassoan Target Version => Slicer 4.7.0
2017-07-24 22:21 jcfr Note Added: 0014943
2017-08-03 00:12 jcfr Status assigned => resolved
2017-08-03 00:12 jcfr Resolution open => fixed
2017-08-03 00:12 jcfr Fixed in Version => Slicer 4.7.0
2017-08-03 00:12 jcfr Note Added: 0015004
2017-08-03 09:07 pieper Status resolved => closed
2017-08-03 09:07 pieper Note Added: 0015005