View Issue Details

IDProjectCategoryView StatusLast Update
0002486Slicer4Core: MRMLpublic2018-05-29 23:46
Reporterlassoan Assigned Toalexy  
PrioritynormalSeverityminorReproducibilityalways
Status acknowledgedResolutionopen 
Product Version 
Target VersionbacklogFixed in Version 
Summary0002486: Remove duplicate saving of node contents to scene and separate file
Description

-----Original Message-----
From: Nicole Aucoin [mailto:nicole@bwh.harvard.edu]
Sent: September 7, 2012 10:38 AM
To: Andras Lasso
Cc: Slicer Developers Mailing List
Subject: Re: [slicer-devel] Duplicate saving of information into MRML scene and files

I'm voting for A, to move the storable plain text data out of the .mrml files and keep them only in their separate files on disk. I did a quick experiment with the color tables, but it didn't work cleanly right away so I pushed it back to after the 4.2 release (if nothing else it shouldn't be writing out the non named colors). The fiducials will be only in files after the Annotations module re-write is finalised.

Nicole

On 09/07/2012 10:20 AM, Andras Lasso wrote:

Hi all,

It seems that the contents of some MRML nodes (e.g., transforms,
fiducials) are saved into both the MRML scene and separate files.

This leads to confusion (it is unclear which file is actually used
when loading the scene) and results in unnecessarily large scene files
and longer save/load times (saving the scene file that contains a grid
transform may take several minutes and the resulting .mrml file size
can be 400-500MB; while saving into an external file takes a few
seconds and the nrrd file size is a few hundred KB).

Possible options to resolve this:
A. Never save these nodes with the scene (similarly to images, which
are always saved as external files) B. Make "Save with scene" an
option in the File Format dropdown list in the Saved Scene dialog C.
Always save these nodes with the scene and just have an "Export to file"
function in the Transforms, Annotation, etc. module

Probably A is the simplest to implement and it would not require any
new design either (it would just work the same way as images).

Andras

TagsNo tags attached.

Activities

alexy

alexy

2012-09-07 07:03

developer   ~0006005

The current convention on loading mrml scenes is that data files
overwrite information in the mrml scene file.

I noticed that in particular vector field transform is stored in the
mrml file that may create huge mrml files. This should be addressed.

nicole

nicole

2012-09-12 11:06

administrator   ~0006045

svn 20977 reduces the size of the mrml scene a bit by not writing out non named colors. Still have to test if this leads to fixing the problems I saw with not writing out colors in mrml at all and just relying on storage node files.

alexy

alexy

2012-09-18 09:02

developer   ~0006124

Last edited: 2012-09-18 09:03

vtkMRMLGridTransformNode stores the displacement values in the MRML. I can remove them, but it may cause compatibility issues if somebody relies on the data being present in MRML

lassoan

lassoan

2012-09-18 10:06

developer   ~0006126

Just write an email to the developer and user mailing list that you plan to remove the storage from the Scene file. If nobody objects for one week then you can go on and remove.

alexy

alexy

2012-09-21 05:56

developer   ~0006157

Removed grid transform displacement field output to MRML file.
Also grid transform can be written as NRRD file now. Reassigning to Jim to verify that it works correctly and can be passed between slicer and CLI's

jcfr

jcfr

2012-11-12 06:51

administrator   ~0007162

From Nicole:

Hi Tiziano,

Yes, the numcolors tag needs to come before the colors tag, due to the
way the color look up table is currently being built. When reading from
file, I parse the file twice to get the hightest index, but the MRML
file is assumed to provide this information. numcolors should be set to
the highest index + 1.

There's a Mantis issue open that will track work to remove the color
information from the MRML file and only read from the separate color
table file, which will be implemented in version 4.3 and hopefully
remove confusion.
http://www.na-mic.org/Bug/view.php?id=2486

A safer way to incorporate your custom color file is to load the ctbl
file via File -> Add Data, then save a scene to see how it looks in
MRML. The order is sometimes important for MRML nodes, check for the
order in which attributes are output in the WriteXML methods of the
nodes in question (or link to the MRML library and use the code directly).

The warnings are from the order of reading the volume nodes, display
nodes and color table nodes from a MRML file or when restoring a scene,
I'm working on detecting the "safe" time to not print out a warning.

Nicole

nicole

nicole

2013-08-27 14:11

administrator   ~0009652

Revisiting this, here's Alex's check in to remove grid transform displacement fields:
http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=21024
For passing to CLI's BSplineDeformableRegistration and TestGridTransformRegistration have them as parameters. Running the latter with fixed.nrrd and moving.nrrd, moving apperas in the Data module under Output Transform. In the Save dialog, I picked Deformation field .nrrd to save the Output Transform. In the resulting mrml file, no extra entries. Tryign to load the Output transform.nrrd file in Add Data, I can only pick a volume file format (or Scalar Overlay or Fibre Bundle). If I load the .mrml scene file, everything loads okay and looks fine.
Doesn't look like there's a CLI that takes a grid transform as input.

Closing the scene, adding a new color file to the scene, and making a copy so there's a new one, then saving... didn't save the mrml file. Checking it to save it, the non empty color entries are saved to both mrml and the ctbl file.

Exit and restart, added 5 fiducials, 2 rulers, 1 ROI, then saved:
Markups fiducial points only saved to disk in one .fcsv file
Annotation Ruler node saved to mrml file and to disk in 2 .acsv files
Annotation ROI node save to mrml file and to disk in 1 .acsv file.
Looading the csv files from disk worked for all of the types.
Loading the scene file worked as well (though the Markups fiducial list didn't get set active on the Markups module node selector).

So: nodes that got moved out of mrml: fiducials, grid transform.
Nodes that did not: rulers, rois, color tables.

nicole

nicole

2013-08-27 15:25

administrator   ~0009655

Did some testing and leaving in the number of colors in the color table in mrml but not writing out the colors seems to be working (aka no crashes like I had last time I tried to fix this). Checked that in as svn 22344[1].

Looking again at the Annotation nodes, I would advise against modifying the Ruler and ROI to not write their control points into MRML.
The Ruler will get moved to the Markups module and will be just written to file via the storage node, not included in MRML.
Alex, if you want to look at the ROI node to take the points out of MRML, please be sure to test scene close, save, load, scene view restore.

[1]http://viewvc.slicer.org/viewvc.cgi/Slicer4?view=revision&revision=22344

jcfr

jcfr

2013-09-10 12:43

administrator   ~0009924

@Alex: Could you update both Nighly / 4.3 module documentation ?

pieper

pieper

2014-03-07 06:40

administrator   ~0011334

Not clear when there will be time to work on this.

lassoan

lassoan

2018-05-29 23:46

developer   ~0015788

Retargeting stale issues to backlog.

Related Changesets

Slicer: 2145-support-for-installing-extension-from-file 80740a1b

2013-08-27 19:23:35

naucoin

Details Diff
BUG: remove color table entries from mrml file

Rely on the file written to disk by the storage node,
reduces the size of the mrml file.
Keep the number of colors in the mrml file so that the
table is properly initialised.

Issue 0002486


git-svn-id: http://svn.slicer.org/Slicer4/trunk@22344 3bd1e089-480b-0410-8dfb-8563597acbee
mod - Libs/MRML/Core/vtkMRMLColorTableNode.cxx Diff File

Issue History

Date Modified Username Field Change
2012-09-07 06:45 lassoan New Issue
2012-09-07 06:45 lassoan Status new => assigned
2012-09-07 06:45 lassoan Assigned To => alexy
2012-09-07 07:03 alexy Note Added: 0006005
2012-09-12 11:06 nicole Note Added: 0006045
2012-09-18 09:02 alexy Note Added: 0006124
2012-09-18 09:03 alexy Note Edited: 0006124
2012-09-18 10:06 lassoan Note Added: 0006126
2012-09-21 05:56 alexy Note Added: 0006157
2012-09-21 05:56 alexy Assigned To alexy => millerjv
2012-11-12 06:51 jcfr Target Version => Slicer 4.3.0
2012-11-12 06:51 jcfr Note Added: 0007162
2013-08-27 12:40 jcfr Assigned To millerjv => nicole
2013-08-27 14:11 nicole Note Added: 0009652
2013-08-27 15:25 nicole Note Added: 0009655
2013-08-27 15:25 nicole Assigned To nicole => alexy
2013-08-30 12:07 jcfr Target Version Slicer 4.3.0 => Slicer 4.3.1
2013-09-10 12:41 jcfr Severity major => minor
2013-09-10 12:41 jcfr Target Version Slicer 4.3.1 => Slicer 4.4.0
2013-09-10 12:43 jcfr Note Added: 0009924
2014-03-07 06:39 pieper Status assigned => acknowledged
2014-03-07 06:40 pieper Note Added: 0011334
2014-03-07 06:40 pieper Target Version Slicer 4.4.0 =>
2017-06-07 23:27 Changeset attached => Slicer 2145-support-for-installing-extension-from-file 80740a1b
2018-05-29 23:46 lassoan Target Version => backlog
2018-05-29 23:46 lassoan Note Added: 0015788