From: slicer-users-bounces@bwh.harvard.edu [mailto:slicer-users-bounces@bwh.harvard.edu] On Behalf Of Ivan Duong
Sent: January 13, 2014 11:03 PM
To: slicer-users@bwh.harvard.edu
Subject: [slicer-users] Extension to Slicer's cropping tool

Hi there,

I am an software engineering student who just started working with the Slicer program at my University. My supervisor requires a tool which will allow us to crop out irregular shaped prisms on the rendered volume; a tool similar to photoshop's 'polygonal lasso tool' but obviously in a 3D way.

Is there such a feature that exists in the application or via extensions?

If not, then I am interested in developing this feature. I have built and compiled the source code, how would you recommend me to go about implementing such feature? Could you give me an estimate to the complexity involved for such a task, especially for a undergrad student; I found it hard to gauge because of the sear size and complexity of the code and my lack of experience.

Thanks so much for your time. Hope to get a reply soon.

Regards,

Ivan Duong.

The main challenge is to draw a polygon on the 3D viewer. Implementing it in the Slicer core might be quite difficult, but it’s possible to implement this as a simple Python scripted extension that relies on already existing Slicer visualization tools:

• The user activates your module and then drops a markup point in the image and drags it around along the region boundary.

• In your module, you observe the changes of the 3D position of the markup point (the good thing is that it already moves the point in one plane, parallel to the image plane) and create a curve (simply add points to a vtkPolyLine and make it more visible with a vtkTubeFilter)

• Once the region is defined, you compute the a 3D mesh from the region boundaries (by projecting the contour points to the near and far plane of the viewing frustum)

• You use the vtkPolyDataToImageStencil and the computed 3D mesh to remove areas from the volume

Another approach to implement this is an editor effect. The user draws a closed contour on one slice, selects the inside/outside option, and pushes the button to apply the contour to all slices of the volume. Draw effect should be a good place to start.

We need a tool that you can use in the 3D view: you select a good viewpoint where the wanted/unwanted regions can be easily separated, then quickly draw a contour; and repeat this many times to get rid of all the unwanted parts one by one.

It would be possible to grab the pixels from the 3D view and copy it into a slice viewer (in real-time) and then use the editor paint tool to define the region in this image. This seems to be is quite similar (or better?) in term of usability and ease-of-implementation, so this could be a good implementation option as well.

Andras, you are right of course - I was mistakenly thinking of 2d application.

Note that the Volume Rendering can handle a mask 3D image (same dimensions than the input volume?) to hide voxels in the rendering.

Hi Guys -

Sounds like an awesome plan - great to see all the ideas flowing. I wanted to mention that we discussed something similar that's needed for the dental application and Bea said that in one of the applications she uses you can just click on a set of points and the software fits a convex-hull polyhedron to it. For example, you could add four fiducials to define a tetrahedron. Or you could take four non-coplanar fiducials and treat them as one corner of a sheared box that defines the ROI (can be quick/simple way to define a 3D shape, plus the fiducials are easy to edit and adjust).

Now that I think about it, this would be a great Editor Effect - essentially the user would select a fiducial list to be rasterized into the current label map. This way the ThresholdPaint and DrawOver options could be used for additional control. Basically we'd want a way to rasterize a convex hull (or even a shrink wrap surface) from a set of points.

-Steve

By the way I did start a prototype at one point of the 'draw and extrude a polygon in the 3D viewer' approach.

https://github.com/pieper/Slicer/commits/threeDEditor

It would be possible to pick up on this, but to be honest I think that defining a surface from a set of fiducials is more promising since it is more controllable for the user. Rather than drawing an outline and applying it, one defines the region parametrically and can go back and tweak the control points until the desired shape is obtained.

A vtkDelaunay3D could be used to generate the convex hull from a set of points:

http://www.itk.org/Wiki/VTK/Examples/Boneyard/Cxx/PolyData/ConvexHullDelaunay3D

Editing the polygon would be very rarely needed, if ever. This is usually a quick "sculpting" operation, not parametric editing. The drawn contour is often not convex, so probably we cannot use the convex hull.

Steve, in the threeDEditor can you actually draw a polygon on the 3D viewer or you just draw in the slice viewer and show it in the 3D viewer?

Andras: It depends on what you want to do - for the craniomaxillofacial use case, defining the volume quickly with a few points is good. But yes, I can see the use case for drawing the extruded prism as well.

Yes, drawing a convex or non-convex outline on the 3D view to define the prism is what I did in the threeDEditor branch.

These are probably the key changes:

here is where the drawn outline gesture is conditional on 2D vs 3D:

https://github.com/pieper/Slicer/blob/threeDEditor/Modules/Scripted/EditorLib/DrawEffect.py#L222

here is the code to rasterize the resulting prism:

https://github.com/pieper/Slicer/blob/threeDEditor/Modules/Scripted/EditorLib/LabelEffect.py#L281-L433

this code basically works but I decided not to invest the time in optimizing it. Since you need to deal with the camera geometry the rasterization is nontrivial. This code steps through the volume along planes perpendicular to the view normal and rasterizes into the label volume. Probably it would be more efficient to draw in the surface of the extruded prism and then flood fill the interior.

It does with with undo/redo, threshold painting, etc so it would be a good place to start for implementing such a tool.

-Steve

Steve,

Thanks for sharing your code. Im sorry if this is an amateur question but how do i compile and test your threeDeditor module?

Hi Ivan -

This code is a little old so it's probably not in sync the current slicer. But probably it's not too hard to test. You'd need to check out the branch and then see if copying the Editor module and EditorLib into the new code would work.

As a way to get started, you can look at the code I linked too in the earlier comment. It's pretty short. These have the mapping from a 2D outline drawn on the viewer into the extruded prism and also has a 'debugGeometry' option for creating a visual representation of the extrusion.

To really understand how to port, debug, and extend this code, you'll need to look at how scripted modules are handled in slicer and how editor effects are created. The links below can help with that. I would suggest getting a little comfortable with that and then looking specifically at the how the threeDEditor code is implemented.

http://www.na-mic.org/Wiki/index.php/2013_Project_Week_Breakout_Session:Slicer4Python

http://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Developers/EditorExtension

Feel free to follow up with questions.

HTH,
Steve

Hi Steve,

I've tried copying the EditorLib and Editor module into the the lastest build of Slicer. When I run the Slicer program the Draw tool no longer works.

I've also cloned the entire threeDEditor branch into a different folder and used cmake to configure and generate a separate Slicer program. I ran Slicer and Draw Tool is working normally like in the original program (drawing on the Slices) but not on the 3D models.

How do I use the code you've created to outline on the 3D model?

Thanks,

Ivan.

Hi Ivan -

In my branch I left it that you could draw on the 3d view the same kind of outline that you would in the 2d view and it would extrude the outline into space. Did you try selecting the draw tool and drawing on the 3d view? You may need to look at the code a bit to see how it's working.

-Steve

Hi Steve,

Thanks for the reply.

For some reason the draw tool no longer works. I havent made any changes to your code yet.

When I select the draw tool the console produces the following error:

Traceback (most recent call last):
File "/home/ivan/MyProject/threeDEditor/Slicer-SuperBuild-Debug/Slicer-build/lib/Slicer-4.3/qt-scripted-modules/EditorLib/EditBox.py", line 337, in selectEffect
tool = effectClass.tool(sliceWidget)
File "/home/ivan/MyProject/threeDEditor/Slicer-SuperBuild-Debug/Slicer-build/lib/Slicer-4.3/qt-scripted-modules/EditorLib/DrawEffect.py", line 101, in init
super(DrawEffectTool,self).init(sliceWidget, threeDWidget)
TypeError: init() takes exactly 2 arguments (3 given)

Am I doing something wrong?

Thanks again.

Hi Ivan -

Did you copy over the LabelEffect code as well? The constructor (init) of the LabelEffectTool should be accepting the optional threeDWidget argument.

https://github.com/pieper/Slicer/blob/threeDEditor/Modules/Scripted/EditorLib/LabelEffect.py#L157

I don't recall all that has changed since the threeDEditor branch was created, but I don't think it's much different so you should be able to get this working. Have you rebased the threeDEditor branch against the current master? If not that would be a good first step. Then if there are still issues you push it to github and we can have a look.

-Steve

Hi Steve,

I've tried to rebase threeDEditor against master, but its not completing the rebasing. It stops with errors such as:

Auto-merging Modules/Scripted/EditorLib/LabelEffect.py
CONFLICT (content): Merge conflict in Modules/Scripted/EditorLib/LabelEffect.py
Failed to merge in the changes.
Patch failed at 0023 ENH: fix camera geometry and code cleanup
The copy of the patch that failed is found in:
/home/ivan/MyProject/Slicer/.git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".

Upon googling the error, I've found that I need to use mergetool to merge these conflicting files. But I'm unsure of when I should use code from the common ancestor, master, or threeDEditor branch.

What would you recommend me do from here?

thanks again for all your help. It's much appreciated.

Hi Ivan -

Yes, that's not unexpected I guess - the threeDEditor branch is pretty old and the master branch diverged since then.

I think if you look at the code that was added in the branch you can get an idea of how it maps into the current master. Eg. you can look at all the commits in that branch that aren't in master and then look at what it would mean to apply the analogous change to the master. In many cases this would just be applying the edit and in others you might need to change the implementation of the edit to match the current conventions.

I haven't had a chance to look at this myself - if you send me the content of the merge conflict I can probably give some ideas about how to adapt the code.

This can help if you haven't already found it:

https://www.kernel.org/pub/software/scm/git/docs/git-merge-base.html

Hi Steve,

I've got the code working now. It allows me to draw onto the 3D volume. and then after applying the drawing it starts to apply the label onto the slices, which is GREAT!

now I can't figure out how to 'remove' what is outside of the label. Is there a button that needs to be pressed or something?

Great progress Ivan!

Regarding removal, I didn't do anything specific for that as part of the threeDEditor, but it should be doable now that the basic 3D interaction is working.

Reading back on some of the earlier notes in this thread, probably what we want to do is create a 'carved' volume that is the intersection of a number of 3D prisms. This carved volume would then serve as a mask for either a segmentation algorithm or for volume rendering.

One way to do this would be to start with any labelmap that includes the structure of interest (say, from a threshold). Then prisms would be ANDed until only the structure of interest remains. We could make this a mode of the apply operation. (Right now the apply operation can do a threshold within the prism, which is also a useful mode).

As Julien mentions, there's an option of masking at the volume renderer level in vtk. I don't think this is currently exposed in Slicer and I'm not sure how to do it (@fintetjul any suggestions?).

As an alternative to using the vtk level, I did a simple module that implements the masking at the MRML level. It allows you to select an input volume, a labelmap mask, a label value, and an output volume. Everything in the output volume is set to 0 except where the labelmap is equal to the label value.

https://github.com/pieper/VolumeMasker

The VolumeMasker module is used to make the renderings at around second 28 of this video.

https://www.youtube.com/watch?v=dfu2gugHLHs

Hi,
i just came across this thread and it seems just like what I need :)

So, does it work already? Perhaps part of the nightly version? (If so, i can't find it).

Z o o m

Push: Nothing new about this feature?

This has been addressed by the VolumeClip extension

See

• https://www.slicer.org/slicerWiki/index.php/Documentation/Nightly/Extensions/VolumeClip
• https://www.youtube.com/watch?v=1mYNwJbE7dQ

The feature is implemented as "Scissors" effect in the Segment Editor (available in current nightly build).