Feature Request: Improved memory management for large cache files

Description

Title:

Improved memory management for large cache files

Category:

Workflow

Status:

Considered

Priority:

Posted By:

Kuubi ( Henri Malkki )

Date Created:

2 February 2017

Description:

I'd like to see improved memory management with larger simulation cache files.

I made a 1000 frames long simulation with 1.2M particles. The sim data takes about 27 GB in PRT format.

Issue 1: Running out of memory when trying to write the cache into an LDR file

I tried to use the LDR format for storing the simulation data, but I watched Max filling my computer's RAM (32 GB) and eventually crashing while trying to write the data. Using Thinkbox PRT files worked, which are created for every frame. Could memory management be improved for the LDR files?

Issue 2: Scenes take a very long time to open with larger cache files

It seems like Lucid tries to read the entire simulation data when opening the scene - it took well over 30 minutes to open the scene mentioned above. Could this be improved? Maybe read the simulation data for each frame after the scene has been loaded? Having extremely long loading times with no indicators of anything happening is not optimal from an end user's perspective.

2 February 2017 #11358
EPHERE	Thank you for the suggestions, both are very good. For Issue 1 I do not understand however how the whole simulation wasn't already in your RAM before writing it to an LDR file. It would have to be inside RAM regardless of whether it gets saved or not. I will need to do some investigation. Marsel Khadiyev (Software Developer, EPHERE Inc.)
2 February 2017 #11359
Kuubi	No idea about the first one. Could it be related to file handling? I can't actually say if the entire simulation was recorded to the LDR file despite crashing - it took only 16 gigs, so I assumed the file would not work. And a quick followup to the Issue 2: the simulation didn't actually load when opening the scene. I got the simulation working by clearing the sim data and loading the PRT files manually, but I was stuck on "show as particles" mode, meshing didn't seem to work. Changing granularity had no effect on the fluid in viewport or in the rendered image. Lucid threw the following assertion after loading the simulation: File: Private\LucidClient.cpp Line: 217 Function: Ephere::Plugins::Autodesk::Max::Lucid::LucidClient::Save Expression: Assert failed: sceneNode != nullptr Also got the following error popup right after loading: "Unbalanced BeginChunk in DerivedObject" After all this tinkering with the cache files I realized it's a lot faster just to run the sim again and forget caching at this point. Hopefully I can render out the required sequences without any crashes.
2 February 2017 #11361
Kuubi	Oh, one more thing to add to the wishlist: make loading simulation data multithreaded if possible. My rig at least seemed to be using only one CPU core when loading the files.

Description

Follow Ups