RSS< Twitter< etc
Monday
Nov212016

Adding And Exposing A Magma Modifier Using MAXScript

Most Krakatoa MX users familiar with the Magma channel editing system use the mouse and the 3ds Max Command Panel features to set up their modifier stack. However, there is a tiny fraction of the users doing TD or TA work that might want to set up Magma and other modifiers automatically as part of pipeline scripts, or custom tools. In fact, some Thinkbox Software products like Stoke include options to add Krakatoa Magma modifiers with a click of a button, for example to delete particles by Age...

While all of the Magma user interface is implemented in MAXScript and the relevant functionality could be found in the freely editable MS files shipping with Krakatoa MX, the process of adding a fully functional Magma node flow with input controls exposed to the Modify Panel is not exactly obvious.

Prompted by a customer support request, here is a look at how a Magma modifier can be added, set up and exposed to the UI using MAXScript.

 

Setting Up The Base Scene

Let's assume that for our testing, we will be adding the Magma modifier to a PRT Volume made from a Teapot primitive. To set up a scene procedurally, all we need to do is create a Teapot primitive, then create a PRT Volume object with the Teapot as the source mesh.

theTeapot = Teapot()
thePRTV = PRT_Volume()
  • Executing the first line will create the Teapot at the world origin.
  • Executing the second line will create the PRT Volume at the world origin.

At this point we can call

show thePRTV

to see a list of all properties exposed by the PRT Volume to MAXScript:

  .TargetNode : node
  .VoxelLength : float
  .UseViewportVoxelLength : boolean
  .ViewportVoxelLength : float
  .InWorldSpace : boolean
  .UseDensityCompensation : boolean
  .DisableVertexVelocity : boolean
  .SubdivideEnabled : boolean
  .SubdivideCount : integer
  .JitterSamples : boolean
  .JitterWellDistributed : boolean
  .MultiplePerRegion : boolean
  .MultiplePerRegionCount : integer
  .RandomSeed : integer
  .RandomCount : integer
  .ViewportEnabled : boolean
  .ViewportDisableSubdivision : boolean
  .ViewportUsePercentage : boolean
  .ViewportPercentage : float
  .ViewportUseLimit : boolean
  .ViewportLimit : float
  .ViewportIgnoreMaterial : boolean
  .IconSize : float
  .UseShell : boolean
  .ShellStart : float
  .ShellThickness : float
false

Here we see that the property we need to set is .TargetNode. We can assign the Teapot to it to convert its volume to a particle cloud:

thePRTV.TargetNode = theTeapot

Looking in the viewport, we can now see the PRT Volume's particles! We can hide the Teapot so it is not in the way:

hide theTeapot

We can also uncheck the option to use a separate Viewport Voxel Length which defaults to 3.0, as opposed to the 1.0 for rendering:

thePRTV.UseViewportVoxelLength = false

At this point, the PRT Volume closely resembles the shape of the teapot, and should have about 48866 particles.

Adding A Magma Modifier To The Stack

The first step is rather trivial - there are two functions in MAXScript for adding a modifier to an object, and both work with Magma modifiers.

The addModifier Function

The addModifier() function has been part of MAXScript since the very beginning, but its functionality changed slightly in the early days of the software. It is very useful because it does not depend on the current state of the 3ds Max UI. It requires two arguments - the object to add the modifier to, and the modifier's instance to add. It also offers a third, optional keyword argument before: which controls where on the stack the modifier is inserted.

First, let's create an instance of the Magma Modifier:

theMod = MagmaModifier()

Now we can add it to the PRT Volume using the addModifier function:

addModifier theTeapot theMod

This will add the modifier to the top of the stack, even if the PRT Volume is not the currently selected object.

The modPanel.AddModToSelection Function, Just For Clarity

The main purpose of this function is to add a modifier to the currently active Modify Panel at the current stack level, while also preserving the sub-object selection. Since the Magma modifier does not require a sub-object selection, and the addModifier() function already offers control over the level on the stack to insert at, we don't really need to pay attention to this option.

 

Loading A Flow From A Preset

One way to set up the nodes of a newly added Magma Modifier would be to load a preset from disk. This would make the whole process very easy, but on the other hand if your custom scripted tool is packaged in just a single MAXScript file, it would mean that script would depend on another file that needs to be distributed. So if you feel comfortable with packaging multiple files and dealing with the question "where is my preset file located so I can load it?", then here are the steps:

Creating The Preset

First, let's create the Magma flow we want to apply to our PRT Volume via MAXScript.

Let's say that we will be performing a simple Push Along Normal modulated by a Noise.

If you have executed all the code so far in an empty scene, you should have the PRT Volume with the Magma Modifier on its stack - simply open the Magma Editor and set up the flow:

  • Press SHIFT+CTRL+P to create a Position Output node.
  • Press SHIFT+P to create and connect a Position InputChannel node.
  • Press the + key on the Numpad to insert an Add operator into the flow.
  • Press SHIFT+N to connect a Normal InputChannel to the second socket of the Add.
  • Press the * key on the Numpad to insert a Multiply operator into the flow behind the Normal InputChannel node.
  • Press CTRL+1 to connect a Float InputValue with a value of 1.0 to the second socket of the Multiply operator.
  • Press the * key on the Numpad again to insert another Multiply between the InputValue and the other Multiply operator.
  • Press CTRL+R to Auto-Reorder the flow.
  • Drag a wire from the second socket of the new Multiply operator and release over an empty area, then select Function, then Noise to connect a Noise operator.
  • Drag a wire from the Position InputChannel node and connect to the first socket (Value) of the new Noise operator.
  • Select the new wire and press the / key on the Numpad to insert a Divide operator.
  • Press CTRL+1 to connect another Float InputValue with value of 1.0 to the second socket of the Divide operator.
  • Rename the new InputValue node to "Noise Scale".
  • Check the Expose checkbox of the InputValue.
  • Select the first InputValue node, rename to "Push Amount" and check the Expose checkbox.
  • Press SHIFT+CTRL+R to bake the auto-reordered positions of all nodes and make them explicit.
  • Change the Push Amount to 10.0 and the Noise Scale to 12.0 and note the results in the viewport.

Now we can save this as a Preset.

  • Click the File menu of the Magma Editor.
  • Select "Save Flow As MAXScript..."
  • In the file dialog, type in the name "PushWithNoise"
  • Click the Save button of the file dialog.

Loading The Preset Using MAXScript

Now that we have our Preset flow, let's see how we can recreate the whole scene including the objects and the modifier using MAXScript:

(--start a new local scope
resetMaxFile #noprompt --this will reset the scene without a prompt

theTeapot = Teapot() --create a default teapot
thePRTV = PRT_Volume() --create a PRT Volume
thePRTV.TargetNode = theTeapot --connect the two
hide theTeapot --hide the teapot
thePRTV.UseViewportVoxelLength = false --switch viewport to render spacing

theMod = MagmaModifier() --create a Magma modifier instance
addModifier thePRTV theMod --add the Magma modifier to the PRT Volume

magmaNode = theMod.magmaHolder --this is the MagmaHolder object that contains the nodes
magmaNode.autoUpdate = true --this will enable the Auto-Update option

--We need to get the default location of Krakatoa MX Magma flows, and get the Preset we just saved:
theFileToLoad = (dotnetclass "System.Environment").GetFolderPath \ 
(dotnetclass "System.Environment+SpecialFolder").LocalApplicationData + \
 "\\Thinkbox\\Krakatoa\\MagmaFlows\\PushWithNoise.MagmaScript" --Then all we need to do is call the following function --that loads the given file into the given Magma flow: MagmaFlowEditor_Functions.loadPreset theFileToLoad magmaNode )--end of local scope

 

Packaging The Preset With The Script Source

In the previous section, we assumed that the Preset file was stored in the default folder where Magma saves its Presets. However, if the Preset will be part of some scripted package that will be distributed to other users, we cannot make such an assumption, unless we either instruct the user to copy the Preset file to a specific location, or write an installer that makes sure the Preset is placed at a specific location.

To make things a lot more self-contained, we can assume that the Preset file will always be located where the master script is called from. So if you instruct the user to copy the files to any folder, and then to run the main MS script, we can modify our code to resolve the path of the Preset based on the location of the main script loading it!

All we need to do is change the line defining the theFileToLoad variable:

theFileToLoad = getFileNamePath (getSourceFileName()) + "\\PushWithNoise.MagmaScript"

By doing this, we ask where the main script is being run from, get the path of that filename, and look for the MagmaScript in the same folder!

Note that if the main script defines a MacroScript to perform the scene and Magma creation, this will result in a copy of the MacroScript body stored in a completely different folder, and would not work with this approach since the Preset would not be found...

 

Embedding A Magma Flow Into a Custom Scripted Tool

As mentioned already, the above approach keeps the Magma flow to be loaded external to the scripted tool. It must be located in the MagmaScripts folder of Krakatoa, or at a specific path the script knows about, or the path could be acquired based on where the script itself is being run from...

However, a more advanced scripter might want to hard-code the content of the flow right there in the source code. This would avoid problems if the external preset file is deleted by accident, or if it cannot be located due to a user mistake, or the MacroScript definition problem mentioned earlier.

Pasting The Flow Definition Into The Script

We can benefit from the fact that Copy & Paste in Magma actually creates a valid MAXScript representation of the Magma flow itself. So all we need to do is select and Copy the nodes in the Magma Editor, and then paste them into our custom script. Alternatively, we could use the content of the Preset file created in the previous section...

  • Open the Magma Editor with the flow created in the previous example
  • Select all nodes by pressing CTRL+A
  • Open the custom MAXScript from the previous example, and replace the last two lines of the script with the content of the Windows Clipboard by pressing CTRL+V
  • Remove the line with the opening bracket (--MAGMAFLOW2--
  • Remove the closing bracket at the end of the pasted text.

If you would execute the script now, the scene would be recreated and the Magma flow would be rebuilt from the code pasted in our script.

However, there are a few small issues:

  • All nodes in the pasted flow are selected, because they were selected when copying to the Windows Clipboard.
  • The exposed InputValue controls will not appear in the Modify panel.

So we need to do some more work here.

Removing The Node Selection

To make the nodes unselected, we can simply delete the two lines of every node definition which look something like this:

magmaNode.DeclareExtensionProperty node0 "Selected"
magmaNode.SetNodeProperty node0 "Selected" true

We can do this for every node.

An alternative solution would be to do Search And Replace for `"Selected" true` and turn it into `"Selected" false`. However, this will result in a longer script as the two lines per node would stay...

Evaluating our script will result in a flow with unselected nodes.

Updating The Exposed Controls

The Preset loading function we used in the previous example handled the updating of any exposed controls. That function is actually defined in the script file Krakatoa_MagmaFlowManager.ms, so it is rather easy to take a look at it and see how it does it...

The loadPreset function definition looks like this:

fn loadPreset theFileToLoad theMagmaNode =
(
	global MagmaNode = theMagmaNode
	theMagmaNode.Loading = true
	fileIn theFileToLoad
	theMagmaNode.Loading = false
	MagmaFlowEditor_Rollout = MagmaFlowEditor_Functions.OpenMagmaFlowEditor magmaNode offscreen:true
	MagmaFlowEditor_Rollout.exposeControlsToModifier()
	destroyDialog MagmaFlowEditor_Rollout
),

 

As you can see, in addition to performing a fileIn() on the preset file, it has three lines dedicated to updating the exposed controls.

  • The first line actually opens the Magma Editor off screen, and stores the resulting rollout value in a variable.
  • The second line calls the function exposeControlsToModifier() in that rollout.
  • The last line simply destroys the dialog made from that rollout.

We need to perform these steps in our code.

Simply copy the three lines from the function definition to the end of our script, just before the closing bracket.

 

The Resulting Script

Here is what the script looks like after all modifications:

(
resetMaxFile #noprompt

theTeapot = Teapot()
thePRTV = PRT_Volume()
show thePRTV
thePRTV.TargetNode = theTeapot
hide theTeapot
thePRTV.UseViewportVoxelLength = false

theMod = MagmaModifier()
addModifier thePRTV theMod
magmaNode = theMod.magmaHolder
magmaNode.autoUpdate = true

global MagmaFlowEditor_EditBLOPHistory = #()
node0 = magmaNode.createNode "Output" 
magmaNode.setNumNodeInputs node0 1 
magmaNode.setNumNodeOutputs node0 0 
magmaNode.setNodeProperty node0 "channelName" "Position"
magmaNode.setNodeProperty node0 "channelType" "float32[3]"
magmaNode.DeclareExtensionProperty node0 "Position"
magmaNode.SetNodeProperty node0 "Position" [1022,0]
--------------------------------------------
node1 = magmaNode.createNode "InputChannel" 
magmaNode.setNumNodeInputs node1 0 
magmaNode.setNumNodeOutputs node1 1 
magmaNode.setNodeProperty node1 "channelName" "Position"
magmaNode.setNodeProperty node1 "channelType" ""
magmaNode.DeclareExtensionProperty node1 "Position"
magmaNode.SetNodeProperty node1 "Position" [182,200]
--------------------------------------------
node2 = magmaNode.createNode "Add" 
magmaNode.setNumNodeInputs node2 2 
magmaNode.setNumNodeOutputs node2 1 
magmaNode.DeclareExtensionProperty node2 "Position"
magmaNode.SetNodeProperty node2 "Position" [882,0]
magmaNode.setNodeInputDefaultValue node2 1 0.0
magmaNode.setNodeInputDefaultValue node2 2 0.0
--------------------------------------------
node3 = magmaNode.createNode "InputChannel" 
magmaNode.setNumNodeInputs node3 0 
magmaNode.setNumNodeOutputs node3 1 
magmaNode.setNodeProperty node3 "channelName" "Normal"
magmaNode.setNodeProperty node3 "channelType" ""
magmaNode.DeclareExtensionProperty node3 "Position"
magmaNode.SetNodeProperty node3 "Position" [602,45]
--------------------------------------------
node4 = magmaNode.createNode "Multiply" 
magmaNode.setNumNodeInputs node4 2 
magmaNode.setNumNodeOutputs node4 1 
magmaNode.DeclareExtensionProperty node4 "Position"
magmaNode.SetNodeProperty node4 "Position" [742,30]
magmaNode.setNodeInputDefaultValue node4 1 1.0
magmaNode.setNodeInputDefaultValue node4 2 1.0
--------------------------------------------
node5 = magmaNode.createNode "InputValue" 
magmaNode.setNumNodeInputs node5 0 
magmaNode.setNumNodeOutputs node5 1 
magmaNode.setNodeProperty node5 "forceInteger" false
ctrl=bezier_float(); ctrl.value = 10.0
magmaNode.setNodeProperty node5 "controller" ctrl
magmaNode.DeclareExtensionProperty node5 "Exposed"
magmaNode.SetNodeProperty node5 "Exposed" true
magmaNode.DeclareExtensionProperty node5 "Name"
magmaNode.SetNodeProperty node5 "Name" "Push Amount"
magmaNode.DeclareExtensionProperty node5 "Position"
magmaNode.SetNodeProperty node5 "Position" [462,115]
--------------------------------------------
node6 = magmaNode.createNode "Multiply" 
magmaNode.setNumNodeInputs node6 2 
magmaNode.setNumNodeOutputs node6 1 
magmaNode.DeclareExtensionProperty node6 "Position"
magmaNode.SetNodeProperty node6 "Position" [602,100]
magmaNode.setNodeInputDefaultValue node6 1 1.0
magmaNode.setNodeInputDefaultValue node6 2 1.0
--------------------------------------------
node7 = magmaNode.createNode "Noise" 
magmaNode.setNumNodeInputs node7 2 
magmaNode.setNumNodeOutputs node7 1 
magmaNode.setNodeProperty node7 "numOctaves" 4
magmaNode.setNodeProperty node7 "lacunarity" 0.5
magmaNode.setNodeProperty node7 "normalize" true
magmaNode.DeclareExtensionProperty node7 "Position"
magmaNode.SetNodeProperty node7 "Position" [462,170]
magmaNode.setNodeInputDefaultValue node7 2 0.0
--------------------------------------------
node8 = magmaNode.createNode "Divide" 
magmaNode.setNumNodeInputs node8 2 
magmaNode.setNumNodeOutputs node8 1 
magmaNode.DeclareExtensionProperty node8 "Position"
magmaNode.SetNodeProperty node8 "Position" [322,185]
magmaNode.setNodeInputDefaultValue node8 1 1.0
magmaNode.setNodeInputDefaultValue node8 2 1.0
--------------------------------------------
node9 = magmaNode.createNode "InputValue" 
magmaNode.setNumNodeInputs node9 0 
magmaNode.setNumNodeOutputs node9 1 
magmaNode.setNodeProperty node9 "forceInteger" false
ctrl=bezier_float(); ctrl.value = 12.0
magmaNode.setNodeProperty node9 "controller" ctrl
magmaNode.DeclareExtensionProperty node9 "Exposed"
magmaNode.SetNodeProperty node9 "Exposed" true
magmaNode.DeclareExtensionProperty node9 "Name"
magmaNode.SetNodeProperty node9 "Name" "Noise Scale"
magmaNode.DeclareExtensionProperty node9 "Position"
magmaNode.SetNodeProperty node9 "Position" [182,255]
--------------------------------------------
try(magmaNode.setNodeInput node0 1 node2 1)catch()
try(magmaNode.setNodeInput node2 1 node1 1)catch()
try(magmaNode.setNodeInput node2 2 node4 1)catch()
try(magmaNode.setNodeInput node4 1 node3 1)catch()
try(magmaNode.setNodeInput node4 2 node6 1)catch()
try(magmaNode.setNodeInput node6 1 node5 1)catch()
try(magmaNode.setNodeInput node6 2 node7 1)catch()
try(magmaNode.setNodeInput node7 1 node8 1)catch()
magmaNode.setNodeInput node7 2 -1 1 
try(magmaNode.setNodeInput node8 1 node1 1)catch()
try(magmaNode.setNodeInput node8 2 node9 1)catch()
--------------------------------------------

MagmaFlowEditor_Rollout = MagmaFlowEditor_Functions.OpenMagmaFlowEditor magmaNode offscreen:true
MagmaFlowEditor_Rollout.exposeControlsToModifier()
destroyDialog MagmaFlowEditor_Rollout
)

Evaluating this script should recreate the scene and set up the Magma Modifier exactly as we want it.

Wednesday
Apr272016

E Unibus Pluram - Partitioning Using Magma

Normally, the Partitioning process in Krakatoa involves taking a live simulation - Particle Flow, Thinking Particles, Stoke, or a PRT object that has a random seed or a modified with a random seed - and saving this simulation multiple times while incrementing the random seed to produce slight differences in the particle distribution.

The Krakatoa PRT Loader object not only supports the loading of these variations from several PRT file sequences as a single particle cloud, but it also has a dirty little secret - it can load them MUCH faster than the equivalent particle cloud stored in a single PRT file!

This multi-threading capability was originally introduced in Krakatoa MX 2.0 mainly to take advantage of the emerging very fast Solid State Drives (SSD), and the even faster Fusion-IO cards (see our Siggraph report from 2011 here). However, the feature was turned on by default and was used all the time, causing some serious complaints from IT departments in large VFX studios as their Krakatoa render nodes suddenly started saturating the network. To avoid this undesired effect, the multi-threaded loading of PRT files is turned off by default since Krakatoa MX 2.2.0, and is exposed as ">Fast PRT Loading" in the "Main Controls" rollout of the UI. To ensure the user is aware that the feature is turned on and potentially dangerous, that checkbutton will always be shown in red when engaged (even if the rest of the UI is customized to have a different color). Also, this option was never exposed in the non-3ds Max implementations of Krakatoa.

How does this option affect rendering? A PRT file contains a zipped binary stream. Its unzipping can be performed only one a single thread. This means that the bottleneck of loading a PRT file is typically not how fast it can be read from the source file, but how fast the CPU can decompress the data. By loading several PRT files on parallel threads, all the cores in a modern CPU can be engaged in the processing of the particles, thus increasing the performance.

This can be very useful when test rendering a scene with a lot of particles - very often the loading time is equal to, or even longer than the time spent sorting, lighting and drawing.

But what can you do if your data is provided in a single large PRT file? Possibly it came from a 3rd party simulation (like RealFlow, Bifrost, Naiad, Houdini etc.), or was just saved originally from 3ds Max without partitioning for whatever reason.

In the following blog entry, we will see how we can split a single existing file into multiple partitions using a simple Magma setup.

 

A Look At The Regular Partitioning

The Partitioning feature of Krakatoa MX was implemented completely in MAXScript as part of the Krakatoa GUI. The Partitioning function looks for scene components (Particle Flow Operators, Thinking Particle Operators, Krakatoa PRT objects and their modifiers etc.) and tries to increment their Random Seed, if they have one. The user can toggle some component categories on and off, or rename individual components to include the token "noseed" to exclude them from being affected. Before saving a PRT file or sequence to disk, the seeds are incremented, and after the saving they are decremented by the same value to restore the initial state. This is repeated for each Partition, where the first Partition uses increment of 0 (no change), the second one used an increment of 1 and so on.

To ensure maximum flexibility to Technical Directors and advanced users, the Partitioning function also includes two slightly hidden (but documented) features:

  1. Each time a new Partition is set up, a custom global MAXScript function Krakatoa_SeedIncrement_UserFunction() is called with the current Partition Index as argument.
  2. A global variable called Krakatoa_SeedIncrement_LastValue is set to that Partition Index before each Partition is processed.

The user can define the custom function to do anything (the function is normally undefined, but its name is pre-declared as a global variable), and the Partition Index global variable can be accessed from any other MAXScript to check out what partition is about to be processed in order to make any other changes to the scene.

In our setup, we are going to take advantage of the Partition Index global variable to adjust a value inside our Magma flow.

 

Selecting All But Every Nth Particle Using Magma

In the core of our approach is the following idea: We want to save every Nth particle to disk, while skipping the rest. The PRT Loader has a "Load Every Nth Particle" option, but we cannot influence which particle it will start with. Let's say we plan to create 10 partitions - we want to save particles with index 0,10,20,30 etc. in the first PRT file, then 1,11,21,31 etc. in the second and so on. As it is often the case with Magma setups, we will simply set the Selection channel of every particle that is not on the list of particles we want for our current Partition, and then delete the selected particles using a Krakatoa Delete modifier.

The actual Magma flow is really simple - we read the Index of the particle, calculate the Modulo by the total number of Partitions, compare the result of the Modulo operator with the current Partition index using NotEqual, and output the result of the logical operator as Float to the Selection channel. The current Partition index is read into the flow using an InputScript operator which executes the following expression:

if ::Krakatoa_SeedIncrement_LastValue == undefined then 0 else ::Krakatoa_SeedIncrement_LastValue

Running the Partitioning feature of Krakatoa with the same total number of partitions as specified in the Magma will save that number of partial files to disk. Loading these files in a PRT Loader will combine them back into the exact same point cloud as the original PRT file. However, if the ">Fast PRT Loading" option is checked, more CPUs will be used to decompress the data in parallel, and the loading will be several times faster.

A .MAX file containing a pre-set scene can be downloaded from the link at the end of this post.

 

How Many Partitions Do I Need?

Of course, it is a good idea to have at least as many Partitions as the number of cores. It might be even worth it having more Partitions than cores. But as we will discuss a bit later, creating more partitions requires more pre-processing time, so usually staying close to the number of cores is a good idea.

Naturally, the actual speed up will also depend on the speed of the storage medium. Loading from a network resource will see less speed up than when loading from a local SSD drive, and a local HDD can even get slower when hammered with parallel demands for multiple files!

Let's look at some benchmark numbers created using a 180 million particles file - the last frame of the Naiad Waterfall simulation provided to us a few years ago by Evermotion. The file contains more channels than Krakatoa actually uses (Position, Velocity, Droplet, EmitId), but they all get to be read from disk. The partitions were saved with exactly the same channels. Rendering was performed on a quad-core Intel Xeon with Hyper-Threading on (8 threads). The file sizes were 4,728,091,907 bytes for the single PRT file, 589,330,049 bytes for each of the 8 partitions, and 295,305,544 bytes for each of the 16 partitions.

Drive Partitions Fast Loading Loading Time Notes
NAS 1 Off 01m 02.759s This is the baseline time, around a minute.
NAS 8 Off 01m 03.337s Sequential loading - file open/close overhead.
NAS 8
On 00m 52.587s Parallel loading - slightly faster.
NAS 16 Off 01m 13.523s Sequential loading - even more file open/close overhead.
NAS 16
On 00m 43.525s Parallel loading - even faster than with 8 partitions.
HDD 1 Off 00m 49.921s Local drive is a bit faster than the NAS.
HDD 8
Off 00m 51.277s Sequential loading - file open/close overhead.
HDD 8
On 01m 16.269s Spinning metal cannot keep up with parallel requests!
HDD 16
Off 01m 02.072s Sequential loading - even more file open/close overhead.
HDD 16
On 01m 19.280s Spinning metal cannot keep up with even more parallel requests!
SSD 1 Off 00m 30.935s Solid State Drive twice as fast as NAS
SSD 8
Off 00m 28.704s Sequential loading, but no file open/close overhead.
SSD 8
On 00m 11.856s Parallel loading - 3x faster than 1 file, 6x faster than NAS.
SSD 16
Off 00m 28.392s Sequential loading, but no file open/close overhead.
SSD 16
On 00m 10.249s Parallel loading - a tiny bit faster than with 8 partitions.

 

Let's look at these numbers.

We can see that loading a single file from the NAS drive takes about a minute, and it benefits only a little from parallel loading because the bottleneck is the network bandwidth. However, Fast Loading actually brings the performance of the NAS drive in line with loading a single file from the local HDD, so that is good news.

In the case of the spinning HDD, we get a nasty surprise - it actually gets slower when trying to load multiple files in parallel, probably because the drive's reading head has to jump around to access them all. If this were a RAID setup with multiple HDDs, the results would have probably looked different. The NAS drive used in this test had 4 HDDs in it, and despite the bandwidth limitations, it outperformed the single local HDD in multi-file loading.

Loading multiple files sequentially (Fast PRT Loading off) increases the access time for both the NAS and the HDD as there are more file open/close operations to be performed.

In the case of the SSD, things look glorious - even loading the single file is twice as fast as reading from the NAS, loading sequentially does not incur a file open/close penalty like on the HDD, and once we enable Fast PRT Loading, it reads 180 million particles in 10 seconds!

 

Other Considerations

Free Proxies For Everyone

As mentioned already, the Krakatoa PRT Loader has a built in feature to Load Every Nth Particle from a file. However, since all particles are packed in the same stream, loading every 100th particle from a file containing one billion particles still requires the loading and unzipping of all one billion particles, where only 1/100 of them will actually go through the modifier stack, transforms, Space Warps, Global Overrides and end up in the memory cache of Krakatoa. In other words, while the 3ds Max viewport will show only 10 million points and won't be overwhelmed with their drawing (esp. in 3ds Max 2015 and higher where drawing of points got much faster), the actual reading time would still be several minutes.

The "Load First N Particles" mode only reads every particle from the beginning of the stream until it reaches the requested count, so it is much faster. But if the particles are ordered in a specific way spatially, then only a small dense chunk of the whole data would be seen in the viewport and would not be representative of the "whole picture".

This is why it is recommended to create a lower-resolution Proxy file by saving the original sequence to a new sequence using the Every Nth mode and a low percentage, for example 1%. This way, loading all 100% of the proxy will show the 1% while reading from a file that is already 100 times smaller, and will be naturally about 100 times faster to load!

The Good News is that creating several partitions of a very large PRT file gives you as many potential proxy sequences - showing any one of them in the viewport while enabling all for rendering will give you both the benefit of a very fast viewport loading, and the benefit of very fast render time loading when the ">Fast PRT Loading" option is checked.

The above screenshot shows 11.2 million particles in the viewport loaded using Load Every Nth Particle mode and 6.25% Viewport percentage from the single file containing 180 million particles. Loading time was 30 seconds from the SSD drive. Expect about a minute from the NAS.

The above screenshot is the result of loading 6.25% of all particles in the single file using the Load First N Particles mode. The loading time was only 4 seconds, but we only get to see the first 11 million particles added by the Naiad emitters to the simulation!

The above image shows only the first of 16 partitions displayed in the viewport. It corresponds to 1/16th (6.25%) of all particles in the simulation, but it was read from its own PRT file at 100% display percentage. Naturally, it took only about 4 seconds to display (reading time from SSD was less than a second, but the data has to be processed, transformed and sent to the Nitrous driver and the graphics card after that).

As you can see, with the one partition shown in the viewport, we get the speed of the Load First N Particles mode, with the display quality of the Load Every Nth Particle mode!

Pre-Paying In Pre-Processing

Saving multiple Partitions from a very large file can take some time to process. Each partition has to read through and actually load 100% of the particles in order to delete the ones that are not needed and save the rest. The more Partitions you save, the longer the processing will take. You can think of this as paying for the speed at render time in advance by spending the time in pre-processing. So this approach makes the most sense in those cases where you expect to re-render the same point cloud many many times, more times than the number of Partitions.

For example, if you have a large PRT file that takes 30 minutes to save 10 Partitions, it would only make sense to create them if you expect to re-load that file at least 10 times during the project (which is quite common).

Tweaking The Channels' Content

While saving the Partitions, you might consider cleaning up any channel data that you don't need, and baking new data that would otherwise cost processing time if calculated each time the particles are loaded.

Normally, once you pick the PRT sequence to partition, you could select the PRT Loader and use the SET Channels List From Selected PRT Objects feature discussed in a another blog post. Then you could remove the Selection channel created by the partitioning Magma, and possibly the Color channel if it only contains the object's wireframe color.

In the test case outlined above, the Naiad simulation contained both Droplet and EmitId channels. If we don't intend to filter particles based on their probability for detaching from the body of fluid, we could remove the Droplet channel when saving. Similarly, if we don't care for the EmitId channel, we could remove it, leaving only Position and Velocity in the Partitions. This would make the files much smaller, and the loading much faster!

The following benchmark results show the difference between loading 8 partitions with Position, Velocity, Droplet and EmitId channels, and 8 partitions with Position and Velocity only. The file size of the PRT files went down from 589,330,049 bytes per partition to 489,470,310 bytes per partition. That means there were 800 MB less data to read.

Drive Fast Loading P/V/D/E P/V Notes
NAS Off 01m 03.337s 01m 01.527s Just a tiny bit faster
NAS On 00m 52.587s 00m 37.456s Significantly faster.
HDD Off 00m 51.277s 00m 40.841s Less data to read, faster in sequential mode.
HDD On 01m 16.269s 01m 02.525s Faster, but still suffering from parallel requests.
SSD Off 00m 28.704s 00m 22.776s Less data to read, faster loading in sequential mode.
SSD On 00m 11.856s 00m 09.765s Less data to read, faster reading in parallel mode, too.

 

On the other hand, if you intend to colorize your particles by Velocity Magnitude using a blue/white gradient, you could set up that in a Magma modifier while saving the Partitions, and bake a Color channel to avoid recalculating the Color procedurally later at render time. Reading one more channel from disk would probably be faster than evaluating a color blend, or even worse, a Gradient Ramp map (if you decided to use a more complex gradient).

Parallel Processing

The creation of the Partitions can be distributed to multiple machines running Deadline, like regular Partitioning jobs. This means that the pre-processing could be performed a lot faster, further lowering the cost of faster loading and free viewport proxy sequences.

 

Download A Pre-set Scene

You can download this .MAX file saved in the 3ds Max 2013 format which contains a PRT Loader with the Magma setup and the Krakatoa renderer prepared. You just need to pick the file / sequence in the PRT Loader, set the output path, set the channels list from the PRT Loader and tweak it as you want, set the frame range or single frame option in the Render Settings, and partition away. Note that the PRT Loader is set to "Load Single Frame Only", you must uncheck that if you intend to partition a file sequence over multiple frames.

 

Conclusion

Even if your data was not partitioned initially, in many cases where the same large file has to be loaded again and again to tweak the rendered look, it makes sense to split it into multiple files using the flexibility of Krakatoa's Magma and Partitioning features. This makes the most sense for NAS, RAID and SDD storage, and also brings side benefits of proxy loading in the viewport, cleaning up unnecessary channels or baking additional channels to disk etc.

Thursday
Apr212016

The Secret Life Of Particle Data Channels - Part 1

A particle in Krakatoa can be seen as a collection of values. These values describe various properties like Position, Color, Density, Velocity, etc. The particle values are stored in Data Channels which Krakatoa can create, modify, save, load, and use for rendering. The same Data Channel design is also used in other Thinkbox products like Frost and Stoke, allowing data to flow between them in various ways.

These Channels are not native to 3ds Max though - they are a core Thinkbox technology that might require a little bit of getting used to when learning Krakatoa, Frost, Stoke etc. But a lot of power and most of the success of these products is based on their existence.

The following feature blog will try to illuminate the life of these Channels, their different birth stories, purposes, and capabilities.

 

A Class Society

When looking at the various Channels that can be used in Krakatoa and other Thinkbox products, it becomes obvious that some Channels are of a higher importance than others - Krakatoa simply cannot live without some of them, others can be useful sometimes, and yet others might be just passing by to help...

The top class of Channels are the ones the Krakatoa Renderer cannot render without. They include Position, Density, and Color.

The next high class of Channels are recognized by the Krakatoa Renderer as carriers of meaningul information, but are not mandatory unless a specific feature has been enabled. They include Emission, Absorption, Velocity and Normal.

Another class of Channels are also required by a specific feature of Krakatoa, but can be allocated manually by the user to allow particles to populate a Memory Channel at render time, or can be turned off and would use a single value for all scene particles. They include SpecularLevel, SpecularPower, Eccentricity and ReflectionStrength.

Yet another class of Channels are required by the Krakatoa Renderer, but are created and managed internally without any intervention or help from the user. An example is the MBlurTime used for Jittered Motion Blur.

At the lowest level of this society are the User-Defined Channels. They can be created, modified and saved to PRT by the user, but Krakatoa does not know what they mean. So they will not be used by the renderer itself, and will not be cached in memory after rendering a frame.

 

How much is this going to cost me?

In its current implementation, Krakatoa 2.x is an in-core renderer. All particles need to be in memory to perform the rendering. As result, every Channel that the renderer needs for a specific shading operation adds a few more bytes to the particle's memory footprint. Multiplying that by the number of particles (which can go into billions) can result in a few extra Gigabytes just for clicking on a checkbutton in the UI to render Environment Reflections, or use Absorption...

The higher class Channels -  the ones the Renderer cannot render without, and the ones that it needs to perform specific operations - have an associated memory cost. And since we don't want the user to guess what the cost is, the UI provides a handy two-way Memory Calculator, and an Active Channels list showing what Channels are going to be allocated, and how many bytes each one will consume.

Once you have enabled the render features like Motion Blur, Emission, Absorption, Reflections etc. that you want to use, you can simply enter the expected number of particles to see if they will fit in your installed memory, or enter the installed memory (minus whatever the OS and 3ds Max already consume) to see how many particles you can squeeze into the available space...

A neat UI trick is the ability to see both the "Main Controls" rollout where most of the options that use Channels reside, and the "Memory Channels" rollout where the cost is calculated. Here is how to do it:

  • Open the Krakatoa GUI (3ds Max menu > Krakatoa menu > Bring Krakatoa To Front!)
  • Expand the "Memory Channels" rollout in the Primary Floater of the Krakatoa GUI
  • Hold down SHIFT and click on the []> icon in the upper right corner of the rollout - the rollout will dock into the Secondary Floater which will appear to the right of the Primary Floater
  • Now you can turn on and off various options like Use Emission, Use Absorption, Ignore Scene Lights, Enable Motion Blur, Jittered Motion Blur, Use Environment Reflections etc. and the Memory Channels rollout in the Secondary Floater will reflect these changes immediately.
  • If you have entered a value in the Memory Calculator, e.g. 100 million particles, the Memory value to the right of it will update dynamically, showing you the memory required based on the changing channels layout!
  • Once you are done, you can SHIFT-click the <[] icon of the Memory Channels rollout to dock it back to the Primary Floater.

 

In most cases, the channels default to using 16 bit instead of 32 bit storage (with one exception we will talk about next). While you can increase the Channel precision by switching any of them to 32 bit via the Memory rollout, in most cases this would be just a waste of memory.

 

The High Society

The one Channel that is absolutely mandatory is the Position Channel. It describes the position of the particle in 3D space at the current time. In the current implementation, the Position Channel is hard-coded to 32 bit float components for X,Y and Z. No particle can exist without a Position. No PRT file should ever be saved without a Position (some versions of the Krakatoa UI allowed this to happen, but it has been fixed in v2.5). In other words, the Position channel is the King Of All Channels!

In theory, to render a point cloud, all you need is a bunch of Positions. In practice, Krakatoa requires at least two more Channels to render - these VICs (Very Important Channels) are Color and Density. However, Krakatoa will gladly accept particles that don't have these channels, and just assign them default values while loading them - White color (1,1,1), and unit density (1.0). However, the result of the render would be merely a black image with a valid Alpha channel, because the Color would not show up without valid Lighting or Emission!

 

Let There Be Light!

Krakatoa is a volumetric particle renderer, and most of the time it is used to render particles that are illuminated by light sources. The particles attenuate their light and can cast shadows on particles bahind them (from the point of view of the light). A portion of that light is scattered into the eye.

The various implementations of Krakatoa handle the lack of light slightly differently:

  • Krakatoa MX for Autodesk 3ds Max (which is the focus of this blog) will show a Warning in the Schematic flow that is a useful tool for debugging the current scene setup (see above screenshot!).
  • Krakatoa MY for Autodesk Maya will show a warning in its Sanity Check rollout. It is the equivalent to the Schematic flow, minus the schematics part. Both can be used to correct the problem semi-automatically.
  • Krakatoa C4D for Maxon CINEMA 4D in its latest version will simply render with the default scene lights  - there is no sanity check implementation in its Krakatoa UI, and too many people have complained about renders being black, so default lights support was added eventually.
  • Krakatoa SR will just render what it is given, it does not have a UI, and it won't complain if no light is available.

There are two cases where particles will render as non-black even when there are no light sources.

If the >Force Additive Mode option is checked in the Krakatoa UI, the Color Channel will be used as if it were the Emission Channel, and the Color (scatter color) will be assumed pure black (0,0,0), resulting in fully emissive rendering without any attenuation of scattering of light. In fact, no lighting will be performed when that option is checked, and the particles will not even be sorted before drawing because A+B is equal to B+A, so the order does not matter at all! The resulting emissive color will be simply added to each affected pixel of the output image, and no Alpha will be generated because Additive renders are supposed to be composited using an Add operation - no blending by Alpha with the underlaying image makes sense there.

The second case is similar, but more finely controlled by the user. If you enable >Use Emission and provide an Emission Channel, it will add light to the particles even if there are no direct light sources in the scene, or even if the >Ignore Scene Lights has been checked to skip the lighting altogether. This is also very useful when the particles have been cached to disk with the very short-named option ">Compute Lighting And Copy Lighting Channel Into Emission Channel When Saving" checked in the "Save Particles" rollout. Loading the particles and enabling >Use Emission while disabling lighting by checking >Ignore Scene Lights will render the particles exactly the way there were baked to PRT, including all inter-particle shadow casting, scattered light, Emission etc.

So in order to see the particles, we need either the Lighting Channel to be populated with data from scene light sources, or the Emission channel to contain non-zero values, or the >Force Addtive option to be checked to make the Color channel act as the Emission channel.

 

The "Butler" Channels

These Channels will only appear when the Renderer needs them for some purpose. For example, if you check the >Enable Motion Blur option, Krakatoa will want to know what the Velocity of each particle is.  This is one of those higher class Channels that are needed for motion blur, and in most cases will be provided by the particle source anyway (be it a live particle system, a procedural PRT object, or a PRT Loader reading from disk). If no such Channel is found for a particle, a zero Velocity of (0,0,0) will be assumed and the particle will simply not motion blur.

However, there is a special Channel that will show up only when both >Enable Motion Blur is checked, and >Jittered Motion Blur is also requested. The Channel is called MBlurTime and adds 16 bits to the memory footprint of each particle. This channel does not have to be generated by the user, in fact it cannot be provided by the user. Krakatoa will allocate it internally to store data needed to keep track of each particle's time as it calculates random Position offsets along the Velocity vector.

 

Where do Channels come from?

As we saw already, some Channels like Position are an inseparable part of the particle. Some Channels can be missing and will be assumed using reasonable default values. Other Channels must be generated explicitly by the user or by the source particle system for Krakatoa to find them and use them. Yet others are not even expected by Krakatoa, but can be created automatically by some objects, or by the user on a whim...

Let's take a look at these different cases and see how Channels are born, and how they pass through their lives in the particle data pipeline. We will start with the Color.

Color Me ... Confused

The Color Channel is one of the most complex cases in the Krakatoa MX integration with 3ds Max. The reason is that there are a few possible color sources in 3ds Max itself, and we have made every attempt to respect them all.

Let's say a particle is being loaded from disk by a PRT Loader, and the PRT file sequence contains a Color Channel already - its value will be assigned to the particle at birth.

  • If the PRT file contains no Color Channel though, then no Color Channel will exist on the Modifier Stack. For exampe, if you add a Magma modifier to the PRT Loader and ask for the Color Channel, you will get a Missing Channel error.
  • However, you can use a Magma modifier to set the Color Channel by outputting a float[3] vector value.
  • On the top of the stack, after the Transforms and WSMs/SpaceWarps have been evaluated, if there is a Material assigned to the object, its Diffuse color will be evaluated and the Color channel will be set to it. So if a Color came up the stack from the PRT file, or a Magma evaluation, it will be totally overwritten by the Material!
  • If there is no Material, and still no Color channel in sight, the Object's Wireframe (display) color will be assigned to all particles when they are loaded for display and rendering.
  • If there is a Global Channel Override Set in the "Global Render Values" rollout writing to the Color Channel, it will overwrite any colors coming from any sources up to this point.
  • Finally, if the >Override Color checkbutton is checked, the solid color or texture map provided under the "Global Render Values" rollout of the Krakatoa GUI will wipe out and replace all colors of all scene particles, regardless of their origin.

Other PRT objects will follow a similar route, except that the initial color value might come from the source object that defines the particle distribution. For examle, a PRT Volume and PRT Surface objects will look for the VertexColor (Mapping Channel 0) data channel of the source mesh, and if it is defined, the particle colors will be taken from it. The PRT FumeFX will read the Color channel of the simulation (if it exists)...

This is similar for particles coming from Particle Flow, except that they will use the Display Color from the Display operator to define the default color. If a VertexColor channel is found in the particles, it will be once again assumed to be the per-particle initial Color source like with meshes, and so on...

Wait, wait! Materials will kill my Colors?

At first glance, you might have a Color Channel coming from a PRT file or another source, passing up the modifier stack, being tweaked by Magma modifiers, just to be mercilessly destroyed by a Material assigned to the object! In many cases, the Material might even be assigned without your knowledge - if you select a mesh object with a valid Material and create a PRT Volume from it using the respective Krakatoa Menu's ActionItem, that same Material will be assigned automatically to the PRT Volume too, and you might not even notice!

But not all is lost.

First, it is very easy to remove the Material assigned to a PRT Volume - just click on the [>] button next to the source mesh's name in the PRT Volume UI, and select CLEAR Material. This will remove the material from the PRT Volume. To copy it back from the current source object, you can use the same menu and go to "Inherit Source Object Settings..." > "Set PRT Volume MATERIAL To Source Object's Material".

Most PRT objects provide a CLEAR Material option in their option menus since Krakato MX v2.5.

Removing a material from other types of objects can be a bit more convoluted - you can either use the "UVW Remove" utility in the 3ds Max Utilities tab > More... list, or select a bunch of objects and type in the MAXScript Listener

$.material = undefined

But what if you want to keep the Material, and still use the Color you so carefully passed up the stack from the birth of the particle? Simply add a VertexColor Texture Map to any map slot of your Material, and the colors of the particles will become part of the Material! Putting the VertexColor map in the Diffuse slot of a Standard Material will propagate the incoming color from the base object and modifier stack through the Material to the Renderer. But you can go further and Blend/Mix/Tint and otherwise tweak that color using all the power of procedural and bitmap-based textures available in 3ds Max!

The main drawback of this approach is performance - evaluating the 3ds Max Texture Maps and Materials is generally slower than using the pure Krakatoa Channels (this is also true for the Maya and CINEMA 4D integration of Krakatoa, too). So while this approach is very flexible, it can influence render times.

Other Standard Material Channel tricks and pitfalls

The 3ds Max Standard Material will not only set the Color Channel though. The Self-Illumination value will set the Emission channel. The Opacity will modulate the Density Channel (this means that the Density will not be overwritten by the Material's Opacity, but will be multiplied to scale proportionally). The Specular Level and Glossines values will set the SpecularLevel and SpecularPower Channels of the particles (if allocated). And the Filter Color under the Extended Parameters will set the Absorption channel.

Passing the incoming Color Channel into the Diffuse component of the Material is easy with a VertexColor map, and the Density will be scaled anyway. But what about preserving the incoming Emission Channel to not be overwritten by the Material's Sefl-Illumination?

Imagine this situation: You create a PRT Volume from a mesh. You add a Magma modifier which sets the Emission Channel to the Absolute value of the Normal Channel of the particles.

You enable >Use Emission, render without any lights and get a beautifully colored self-illuminated point cloud:

Then you assign a Standard Material because you want to set the Color Channel via the Diffuse color, or modulate the Density via the Opacity slot using various textures. You render, and your Emission Channel is completely gone because it was overwritten by the Self-Illumination of the Standard Material which defaults to zero!

How many ways can you find around this problem? Thankfully, there are a bunch...

You can add a VertexColor map to the Self-Illumination map slot. Set it to Map Channel 2.

Now open your Magma Editor and instead of writing to the Emission Channel, write to Mapping2.

Render - your particles will be back to their old colorful glory!

The obvious alternative is to avoid using the Standard Material. The dedicated Krakatoa Material is much more lightweight, and has a checkbox to control whether to set any of the supported Channels. So if you want to set the Scattering (Color) and Absorption Channels, or the Density Channel, but you don't want to overwrite the Emission, you can make sure the right checkboxes are toggled, and you will be safe.

Here is the above example, but with a Cellular map scaling the incoming Density values, times 3.0:

The Magma was changed again to output to the Emission Channel. The rendered image shows the Emission was not affected by the Krakatoa Material because the Emission checkbox was unchecked:

Finally, the Magma Modifier provides an InputTexture node which can evaluate any Texture Map (or a whole tree of Texture Maps) at any point on the Modifier Stack to produce the same results without the need for a Material assignment!

Apropos Magma Modifier...

The original name of the Magma Modifier upon its introduction in Krakatoa 1.5 was, not surprisingly, KCM, short for Krakatoa Channels Modifier. However it was renamed in Krakatoa MX 2.0 because Magma is shorter and a lot more colorful (pun totally intended) for marketing purposes.

When you assign a Magma Modifier and open the Magma Editor, the Output and InputChannel nodes provide a long list of Channel names. These lists are partially hard-coded and also include any Custom Channels defined in the Krakatoa GUI > "Save Particles" rollout, but they are offered mostly for convenience. While they include most Channels the Renderer knows about, plus most Channels used in Particle Flow, FumeFX, Naiad, Stoke, Frost etc., the Magma editor lets you type in literally any valid Channel Name. If the Channel Name is entered in an Output node and that channel does not exist yet, a new Channel with that name will be created. If the Channel Name is entered in an InputChannel node, that Channel must exist on the Modifier Stack below the Magma Modifier.

This lets you create Custom Channels on the fly and pass their data up the stack between Magma Modifiers.

Since the Renderer does not know anything about these Custom Channels, they would not have any effect on the final rendering, and cannot be used to pass data to the renderer or to a Magma-based CustomData Render Element (if you try to read such a Channel into a Render Element, a popup will tell you to use one of the many Mapping Channels instead!). However, Custom Channels will go as far up the pipeline as the Global Channel Override Sets (which are of course also based on Magma). This gives us yet another way to work around the above-mentioned problem of a Standard Material overwriting the Emission channel!

In the Magma Modifier on the PRT Volume, set the flow to output to a custom Channel called Test1.

Open the "Global Render Values" rollout of the Krakatoa GUI, and press the "Create New Global Override Set" button. Set the flow to read the Test1 custom channel defined on the PRT Volume object, and output it as Emission.

Even if you would assign a Standard Material to the PRT Volume, the values will go around it via the Test1 Channel and the renderer will get its Emission and render it correctly!

Obviosuly, since the Global Channel Overrides are applied to all particles in the scene, using the above approach would require a Test1 Channel to be assigned to every particle source that will be rendered. So it is not very practical compared to the other workarounds, but technically it works.

To Be Continued...

In the upcoming second part of this blog topic, we will look at the creation and life of the Density Channel which is almost as complicated as the Color Channel...

Wednesday
Apr132016

Setting The Save Channels List With One Click

The ability to save particles to file sequences using the Thinkbox PRT file format has been a cornerstone of the Krakatoa workflow. Historically speaking, the first in-house version of Krakatoa used by Frantic Films to render the billion particles per frame in Superman Returns back in 2006 was a stand-alone command line renderer that could render only PRT files.

When Krakatoa was first integrated into 3ds Max, the caching of particles and the creation of Partitions in a first pass to be combined and rendered in another pass was already a great production-proven workflow. Once the PRT Loader object was added, it allowed particle assets to be reused, combined, transformed, retimed, deformed, culled, and their channels to be modified using Magma once that technology came along.

In the previous Krakatoa MX Feature Blog, we looked at the various ways to populate the File List of a PRT Loader. However, before the PRT Loader can load anything, a PRT sequence needs to be written to disk first.

A PRT file supports arbitrary channels, and Krakatoa recognizes a number of pre-defined names for various purposes. For example, the Velocity channel can be useful for both retiming the particles in the PRT Loader and creating Motion Blur effects, the Color channel controls the light scattering, the Density controls the contribution of the particle to the density of the volume being rendered, and without a Position channel there would be no particle to start with.

But there are a lot more channels available from Particle Flow systems, FumeFX simulations, various procedural Krakatoa objects, and the user can even make any number of custom channels on top of that.

The saving workflow involves switching Krakatoa to "Save Particles To File Sequence" mode, defining the output path, and setting up the list of channels to be saved.

The Save Particles lists and the source of confusion

If you look at the "Save Particles" rollout of the Krakatoa MX GUI, you might get the feeling of "Now what?" - there are so many channels that could be saved to disk, and it is a tough call to make the decision which ones to include and which ones to skip.

Every channel that contains actual data will cause the PRT file to grow bigger, which is never a good thing.But if you forget to save a useful channel that will be needed later in the PRT data processing, it would mean you would have to re-save the particles, costing you precious time.

A channel that contains no data will save a default value which will be compressed pretty well when the PRT file is saved. However, it could be confusing to other artists using the asset as they might expect useful per-particle values from that channel.

The "Save Particles" rollout proposes a default list of "Save Channels" when Krakatoa it initialized: Position, Velocity, Density, Color, Normal, ID. These are the Usual Suspects, but in many cases saving all these channels would be an overkill. For example, if you are saving Particle Flow particles, the Density value will most probably be 1.0 for all particles, so including an explicit channel for Density would be a waste. But if you are saving a PRT Volume to disk, each particle would have a unique Density value based on the density distribution within the mesh. Also, a PRT Volume has no valid ID channel, but a Particle Flow system always has one, and it can be very useful to save.

On the left side, the "Do Not Save Channels" list proposes a wealth of information, but not every particle has those channels. For example, the SignedDistance channel makes sense when saving a PRT Volume, but does not make sense for most other particle sources, unless specifically created using a Magma modifier. Fire, Fuel and Temperature are typical FumeFX channels so unless you are saving a PRT FumeFX to a PRT sequence, adding those channels to the Save Channels list does not make any sense either.

 

The Magical Solution hiding in plain sight

The [>>] button between the two Channel Lists lets you define presets of typically used channel combinations. It also lets you set the current Save Channels list as a startup preset for fresh instances of the Krakatoa renderer, so if you don't care for the Density, Color or Normal in your typical workflow, you can remove them from the list, and set the new list as the Default.

But there are two more entries on that menu that are what this blog entry is all about: "SET Channels List From Selected PRT Objects" and "APPEND Channels From Selected PRT Objects". These options let you populate the Save Channels list with just the right channels by looking at the channels of the objects in the current scene selection.

For example, let's say that you have created a PRT Volume from some mesh, probably added some modifiers to it, and you want to bake all channels generated by this PRT Volume to a PRT file sequence. In theory, you could open the Particle Data Viewer from the Krakatoa menu, take a look at what channels exist in the PRT Volume, and add them manually to the "Save Channels" list one by one.

But by selecting the PRT Volume and simply clicking on the "SET Channels List From Selected PRT Objects" menu option, the list can be populated automatically!

 

Destruction is always easier than Creation

Now that the list is populated with every channel that exists in a PRT Volume, you can easily remove any channels you don't feel are needed. For example, the Velocity channel would only be meaningful if the source mesh of the PRT Volume has animated deformations (the particles would create Velocities based on the deforming mesh). If you don't have such deformations, you can double-click the Velocity channel to send it to the Do Not Save list. The Color channel will usually contain the wireframe color of the mesh, unless a Vertex Color channel was acquired, a Material was assigned, or some fancier Color channel was set up using a Magma Modifier. So you can banish the Color channel to the left list, or keep it depending on the case. The SignedDistance contains the distance to the closest point on the source mesh surface with a negative sign (because it is inside the volume), while the SignedDistanceGradient contains a vector that points in the direction of the increasing SignedDistance. In most cases, using the Normal channel would give us a similar direction, so we might want to remove the SignedDistanceGradient...

So with a click to populate, and a few double-clicks to decimate the list, we are now ready to save the existing channels to PRTs.

This of course works with any number of selected PRT objects - you can have a PRT Surface, a PRT FumeFX and a PRT Loader in the scene and you can build the combined list of all their channels by simply selecting them and picking the option from the menu. Of course, any channels that are available in one object but unavailable in another will contain zero values for particles that don't have that channel, but it is still a very convenient way to figure out what might be worth saving.

You can also use the Append... version of the feature to keep on adding channels from other objects after a list has already been started. Exicting channels will not be duplicated, so it is very useful to add just the ones missing.

 

How about Custom Channels?

Normally, in order to save a custom channel to a PRT file, you need to define the custom channel via the "Save Particles" rollout's "Edit Custom Save Channels (Marked with * on the above Lists)" controls.

For example, if for some reason you used a Magma Modifier to create a custom channel called "Bobo" and want to add it to the Save Channels list to bake it to PRT, the standard workflow would be to also enter "Bobo" in the custom channels edit field, set the channel type and Arity (e.g. float16[3]), press the [+] button on the left side to add it to the left list, and then move it to the right list.

This custom channel definition will persist between sessions via an INI file. That channel will always appear on the Do Not Save Channels list in each new Krakatoa MX session. It will appear on the channel lists of the Magma Editor's InputChannel and Output nodes. In other words, it will be relatively permanent, until you select it from the list and hit the [X] button on the right to delete it.

Obviously, after a while the channel list can get polluted with various custom channels that were used just once in a project and then left behind.

Thankfully, the "SET Channels List From Selected PRT Objects" menu option does not require a custom channel definition!

Let's say that you add a Magma modifier to the PRT Volume described above, and set an Output node to write the Normal channel multiplied by the SignedDistance into the new channel called "Bobo".

Select the PRT Volume, pick the "SET Channels List From Selected PRT Objects" menu item, and the channel Bobo : float16[3] will appear on the right list, without an * denoting a custom channel definition.

Saving with this setup will save the new channel into the PRT file. Saving the 3ds Max scene will keep that channel on the "Save Channels" list.

However, this channel is a bit like a vampire. It can only live on the right (dark?) side of the UI. If you try to move it to the left "Do Not Save Channels" list, it will vanish, because it is not a hard-coded factory default and is not stored in an external INI file. If you switch to another renderer and then reassign a new Krakatoa instance, it will be gone. If you restore the previous Krakatoa instance using the Krakatoa menu though, it will be back...

 

Limitations

As the menu option implies with its name, only PRT objects (like PRT Loader, PRT Volume, PRT Surface, PRT Hair, PRT FumeFX, PRT Source, PRT Maker, and the Stoke MX PRT Field) are currently supported by this feature.

Selecting a Particle Flow will not provide any insight into its channels - the channels list of Particle Flow in the Particle Data Viewer is hard-coded, since all channels exist in a Particle Flow system all the time.

Wednesday
Apr062016

The Many Roads to Populating the PRT Loader's File List

The PRT Loader object of Krakatoa MX is a key component of the system - it lets you load, retime, transform and edit not only PRT files saved by Krakatoa itself, but also 3rd party files like RealFlow BIN or PRC, LiDAR point cloud data in various formats like LAS, LAZ, E57, PTG, PTS, PTX, etc., and even text files written by any application that can output a comma-separated list to CSV, XYZ or even TXT.

The PRT Loader offers a file list control which can hold one or more file sequences. To simplify your life (or to make it more complicated, depending on your point of view), Krakatoa MX offers a lot of different ways to populate the file list with files or file sequences.

In the following feature blog entry, we will take a look at most of these features.

 

Loading a single file sequence, the manual way

Loading a single animated sequence is easy - create a PRT Loader through the Create panel of 3ds Max, click the Add Files... button above the file list, navigate to the folder where your file sequence is stored, and just pick any of the numbered files in the sequence. The frame number will be automatically replaced on each frame with the correct frame number to load, so by picking only one file, you have picked a whole lot of files to be loaded over time.

In one case though, which file you select will be important.  If you intend to load a single frame from a sequence and use that file on every frame of the timeline by checking the "Load Single Frame Only" checkbox, then you should pick the exact frame you want to be loaded. With this option checked, the frame number will be left As Is and will not be replaced with the current time's frame number.

 

Creating the PRT Loader and opening the File Dialog with just one click

Krakatoa MX offers an ActionItem (implemented by a MacroScript) that is available immediately via the Krakatoa menu in the 3ds Max menu bar.

It can also be added manually by customizing a toolbar, a keyboard shortcut, or QuadMenu.

Simply calling that ActionItem will let you click in the viewport to place the PRT Loader's icon, or if you hold SHIFT while calling the ActionItem, the PRT Loader will be placed at the world origin to ensure that the particles appear in world space exactly where they were saved from.

However, in 99.9% of the cases, creating a PRT Loader means that you want to load some file sequence with it. So when you use the PRT Loader ActionItem instead of the Create tab of the Command Panel, the File Dialog will pop up automatically. This way you save the click on the Add Files... button (and those clicks tend to add up really fast in production!)

This behavior is not mandatory though - if you are annoyed by the File Dialog popping up automatically each time you create a PRT Loader via the Krakatoa menu, you can simply turn that off in the Krakatoa Preferences dialog. With the Krakatoa renderer already assigned, select the "Open the Krakatoa Preferences dialog" menu item in the Krakatoa menu. In the "User Interface and Interaction Preferences" rollout,  look for the "When creating a New PRT Loader" option, and set it to "Do NOT Open File Dialog After Loader Creation".

 

Navigating Sub-Folders can be a waste of time, too...

Ok, so we can save a click to the Add Files... button, but then we still have to navigate the folders to find our file sequence. That can be a lot of additional clicks!

Well, it depends on how your files are organized. If you have a central location or a dedicated drive where you store most of your PRT sequences, for example in project-oriented sub-folders, Krakatoa can help you get there faster!

In the same Preferences dialog mentioned above, you will also find an option called "Default Path For New PRT Loader", with a file picker button, a text field, and a History button. You can pick any folder on your drive where you would like to start navigating from when you create a new PRT Loader. Or you can click the History button and pick any folder that you have saved to before (Krakatoa tries to remember everything you ever did, after all). Once you have set that path, creating new PRT Loaders via the ActionItem will open the File Dialog at that location, except when...

 

You just saved some PRT files and you probably want to load them...

The Krakatoa GUI has a "Save Particles To File Sequence" mode which is probably the main source of PRT sequences on your disk drive. So if you just saved a new particle file sequence with Krakatoa and want to load it, wouldn't it be nice if the PRT Loader helped you with that, too?

Indeed it does by default. When the Preferences dialog specifies the option "Open File Dialog After Loader Creation At Current Save Path, If None - At Default Path", the PRT Loader will first check if a valid path is defined in your Krakatoa GUI, and it will open the File Dialog at that location. Otherwise, it will use the Default Path we discussed above.

If you want to override this behavior, you can hold down the CTRL key while the file menu is being opened. For example, you can hold down CTRL+SHIFT while using the menu ActionItem, or hold down CTRL while using the ActionItem with manual placement of the object in the viewport, or hold down CTRL while clicking the Add Files... button of an existing PRT Loader at any time. If the CTRL key is down when the file dialog opens, the Default Path will be used instead of the Save Particles path! Of course, you can override this behavior permanently by selecting the alternative option "Open File Dialog After Loader Creation At Default Path" to ignore the Save Particles output path and always open at the Default Path.

You can even get the Save Particles path at any time after the PRT Loader has already been created! In the [>>] options menu, there is the option "Add Save Particles Sequence". Selecting this option will add the output path (if any) defined in the Krakatoa GUI > "Save Particles" rollout to the PRT Loader's File List. If there is no valid path, nothing will be added.

 

You just loaded a file sequence, now you need to load another one from the same folder...

In some cases, you might want to load several files or file sequences from the same location. For example, you might have a main PRT sequence for rendering, and a second proxy PRT sequence containing only a fraction of the points for very fast viewport previews. That sequence could be in the same folder, or in a sub-folder.

However, the File Dialog does not allow the picking of multiple sequences at once (we will look into how to add multiple files at once a bit later). But if you have already created a PRT Loader and picked a file sequence from a project folder that is a few clicks away from the Default Path, opening the File Dialog by pressing the Add Files... button would mean more wasted time, right?

Thankfully, the PRT Loader will override the Save Path / Default Path behavior if a file sequence is highlighted in the File List control in its UI! So if you have already added a file sequence to the PRT Loader and you need to add another one that is in the same (or in a very close by) directory, you just need to highlight that existing file sequence entry on the File List, click the Add Files... button, and the File Dialog will open directly in the same folder!

If more than one file sequence is highlighted though, that selection will be ignored and the default behavior specified in the Preferences dialog (open at Save Particles path, at the Default path, or in the current folder) will be used instead.

 

You can't pick the same File Sequence more than once, except ... you can!

The PRT Loader will refuse to accept the same file more than once on its File List. Having the same file twice would mean all particles would be simply duplicated, wasting memory and doubling the Density without any visual benefit. However, this is only enforced when the exact same file name (including the frame number) is selected. So if you already have "c:\temp\particles_0000.prt" on your list, you cannot add that file again, but you can add "c:\temp\particles_0001.prt" without a problem. This allows you to enable "Load Single Frame Only" and have multiple frames of the same sequence loaded as a single point cloud.

 

But adding hundreds of files would mean a lot more clicking, right?

As mentioned before, the File Dialog of the PRT Loader opened by the Add Files... button allows the picking of only one file at a time. But that is not the only way to add files to the File List, which is the whole point of this blog entry ;)

First, there is the File Sequence Manager dialog accessible through the (slightly incorrectly named) "Open File Sequence Editor" menu item in the [>>] button's content menu of the PRT Loader.

In this dialog's File menu, you will find two options called "Add Files From Folder..." and "Add Files From Folders Recursive...". The former will let you pick a single folder and will load every file sequence found in it. The latter will do the same, but will also search all sub-folders of the selected root folder, and add all file sequences found in all of them. So if you have a folder containing several file sequences, whether they were created by Partitioning or by saving multiple times with completely different names, you can use this feature to quickly add these file names to your PRT Loader.

 

Did I just mention Partitions?

You are probably already aware of the fact that selecting a sequence that is part of a set of partitions will present you with a specialized dialog that lets you add some or all of the partitions, even ones that do not exist yet (but might be just being saved by some network render nodes under Deadline's control to be available shortly for rendering).

This only works of course if at least one file of one partition has already been saved to disk - after all, you need something to pick in the File Dialog for the Load Partition dialog to know what to display.

However, Krakatoa will let you populate the PRT Loader's File List with all partition sequences even before a single one of them has been saved to disk! Simply click the [>>] options menu button and select the "Load All 'Save Partitions' Sequences...". The PRT Loader will take the Save Particles output file defined in the Krakatoa GUI, apply the naming conventions for Partitioning by using the Partition Count value found in the "Partitioning" rollout, and will add all file sequences to the File List.

Note that only sequences that are not on the list yet will be added. For example, if your Partition Count is 10, and you select this option to add the 10 partitions to the File List, then select Partition number 07 and press the Remove... button to delete it from the File List, and then select the option from the menu again, the 7th partition will be added to the end of the list, but the other 9 partitions will not be duplicated (we already know that duplication is generally a bad idea).

 

Saving a File List today might help you tomorrow...

Another feature of the File Sequence Manager is saving and loading of File Lists.

A File List is a preset file written to disk that can be created from one PRT Loader and loaded later in another PRT Loader to recreate the same content. It contains a comma-separated list of values, where the first column in the file name, and the second value is the loading mode. A value of 3 means load in the viewport and in the renderer, 2 means load in the renderer only, 1 means load only in the viewport, 0 means do not load anywhere. If you are a programmer, yes, the first two bits of a byte are being used as flags, and their state is shown in the File List as a prefix ("v", "r" or "--"), and as checkboxes labelled "Viewport" and "Render"...

Obviously, in order to save a File List preset to disk via the File Sequence Manager, the File List must first be populated. Of course the content of a File List file (the .KFL extension comes from Krakatoa File List) is so simple, you could write one by hand, or using MAXScript. But why overcomplicate things when a slightly hidden feature of the Windows Explorer and the good old Windows Clipboard can help us get there without writing a single line?

 

Copy And Paste that file list, Like A Boss!

You might be aware of the fact that if you select a file in Windows Explorer, holding SHIFT and right-clicking the file will add a new "Copy as path" option to the context menu. Selecting it will copy the path of the selected file to the Windows Clipboard, and you can paste the file name anywhere.

But you might be surprised to learn that you can select ANY number of files in Windows Explorer and do the same to create a file list in the Windows Clipboard that the PRT Loader will gladly accept!

The content of the Windows Clipboard will simply have each selected file in a separate line, with quotation marks around the file name to ensure spaces are handled correctly. The order of the files will be affected by the order in which they were picked, for example the the following screenshot I picked the 01 partition first, then SHIFT-clicked the 10th partition, which caused it to be listed as the first file on the list.

Now all you have to do is click the [>>] button in the PRT Loader's UI and select "Paste Filenames From Windows Clipboard" - the File List will be populated with the file names you selected in the Windows Explorer!

However, the loading mode flag will be set to 3 (load in viewport and renderer), so it might be a good idea to first turn off the Viewport display or enable the Viewport Limit as all pasted files will try to load and could slow you down...

 

What about the reverse - open the Windows Explorer at the path of an already added file?

That's easy - simply double-click the file name in the File List, or, if you are working in the File Sequence Manager dialog, in its list! A new Windows Explorer will pop up immediately and reveal the location the file came from.

 

And when everything else fails, there is always MAXScript...

Obviously, for the hard-core TD or TA in a studio, setting the File List via scripting is often a necessity. For example, if a file sequence was generated by a script using the MAXScript Particle Output Interface exposed by Krakatoa, the next logical step would be to create a PRT Loader and populate its File List, all through the same script.

Obviously, a MAXScript can set any filename from any path, even if the file does not exist yet. This is a great power, and with great power comes great responsibility. If you are a developer setting file sequences in a PRT Loader, you should remember that there are always TWO properties that need to be set for the PRT Loader to function properly - for each file name, there is a matching flag defining the loading mode. We already saw this when discussing the File List saving and loading in the File Sequence Manager - the two bits of the mode value control the viewport and renderer loading of the file.

In the following example, a file sequence with 101 frames saved from Particle Flow is loaded into a PRT Loader in "Load Single Frame Only" mode, so that all 101 frames are loaded together, showing all particles at all times.

theLoader = KrakatoaPRTLoader() --create the PRT Loader
theLoader.percentViewport = 100.0 --set viewport display to 100%
theLoader.loadSingleFrame = true --set to load single frame only mode
theFileName = @"C:\Temp\partitions\Shockwave\shockwave_0000.prt" --this is the first frame
--collect all 101 frames:
theFileList = for f = 0 to 100 collect FranticParticles.ReplaceSequenceNumber theFileName f 
--set all  files to be visible in both viewport and renderer:
theFlagsList = for f in theFileList collect 3 
theLoader.fileList = theFileList --assign the file list
theLoader.fileListFlags = theFlagsList --assign the flags
max modify mode --switch the command panel to Modify mode
select theLoader --select the PRT Loader

 

Conclusion

There are many ways to populate the File List of a PRT Loader, from simple UI operations through Windows Clipboard Copy&Paste to hardcore scripting. Knowing about all of these options can save you a bunch of clicks and a lot of headache during production crunch times...