Under the prefenceses GPU rendering is selected for volume rendering options, plus 4GB GPU RAM is specified. However, when I switch to volume rendering module and look at the advanced tag, gpu ram shows as 0MB. I manually switched it to 4GB prior to rendering. Made no difference. Still crashed.

I made sure slicer-real-app.exe is added to the nvidia control panel to make sure it is using the nvidia chipset. While

Quadro cards often have compatibility problems (probably because so few people use them and report errors), so if you have the choice, switch to GeForce (you get much better performance for the same money anyways).

It may still be a Slicer issue though. Could you try a nightly build from a month ago?
https://download.slicer.org/?offset=-30

Also, If your GPU has 4GB RAM then don't allow the volume renderer use 4GB. Try 1-2GB instead.

It is a laptop, I don't have a choice to change it. Same dataset works perfectly fine with stable version with specified 4GB GPU RAM on the same system. I will give it a try with an older nightly.

Latest stable version of Slicer used VTK7 and its ancient OpenGL API, which had good hardware compatibility (ran on 15-year-old hardware). However, VTK switched to more recent OpenGL APIs, which offer much better performance but there are more potential compatibility issues, as they use more enhanced features.

Is your BIOS and graphics card driver up-to-date? Does volume rendering in ParaView work for you? Does CPU volume rendering in Slicer work?

It is a laptop from 2014 (dell precision M6800), bios is the latest for it. I will try updating the Nvidia driver, though I think the existing driver is only a 6 months old or so (v390.77). CPU rendering works fine (but slow). I don't have paraview.

You can download paraview from here: https://www.paraview.org/download/. It uses the same rendering backend as Slicer. If it crashes, too then the error is most likely in VTK or lower level.

In the future, I would recommend to stay away from Quadro cards, unless you only use applications that are certified to be compatible with them. Quadro cards are optimized to work with a list of 100 or so "professional" applications and often not work very well with others.

I tried to reproduce, and although it gets slow sometimes, there is no crash. I have a GeForce 1060 3GB.
What are your settings? VTK GPU Ray Casting I assume. What about Quality (Adaptive/Normal/Maximum)? Did you use any transfer function other than the default? What does it mean "slide the transfer function down to about 5"?

I reproduced it on a desktop with TITAN (original one with 6GB) on windows 7. It has the latest set of drivers from NVIDIA.

I use VTK GPU ray casting with adaptive quality (8fp). I modified the default transfer function such that the low point on scalar opacity function is to 5 and the high is set to 6 (so a very short ramp). Then try interacting with the volume extensively zoom in, zoom out, spin etc. It works about 5-10 secs, and then driver crashes (on Windows 7) or get a TDR (on windows 10).

Thanks for the details! When I did what you suggested Slicer just quit. It wasn't even a crash. I'm not sure what could be possibly done to diagnose the exact reason.

On windows I do get an error, but not from Slicer from Nvidia driver (both on Windows 7 and 10). If you are on Linux/mac may be syslogs may help.
Again same dataset works perfectly fine on 4.8.1 with the same settings.

I could reproduce the crash, too. Details were described in the system logs: it was a TDR (http://nvidia.custhelp.com/app/answers/detail/a_id/3633, https://docs.microsoft.com/en-us/windows-hardware/drivers/display/timeout-detection-and-recovery). It seems that the GPU is simply too busy because of the huge workload.

I've "fixed" the issue by increasing TDR delay to 10 seconds (by adding TdrDelay=10 to registry https://docs.microsoft.com/en-us/windows-hardware/drivers/display/tdr-registry-keys) and setting the image's data type to unsigned char, and using Adaptive or Normal setting. I could still get TDR on maximum quality.

While searching for solutions for TDR error on the web, I've found that many scientific software has this error and the recommended solution is to increase TDR delay, so this seems to be a common solution.

However, I think the GPU raycast mapper could be improved to reduce the chance of this error.

1. When setting GPU memory usage limit to a lower value then the volume supposed to be downscaled in the mapper to fit in that size. I don't see an effect of maximum GPU memory size, so maybe this mechanism is broken in Slicer or VTK.

2. vtkGPUVolumeRayCastMapper could be changed to perform processing in smaller batches. However, implementing this is probably not trivial, it could slow down rendering, should not be necessary for commonly occurring volume sizes, and there are multiple workarounds (increase TDR delay or downsample volume), so I don't think this will be very high on the priority list of VTK developers. We should still report this, so that developers are aware of this and users who encounter this issue get a clue about what is happening and what can be done.

Entered a VTK issue: https://gitlab.kitware.com/vtk/vtk/issues/17342