Sandboxing: Answers to Questions You Didn’t Know You Had
Version: Deadline 8.0
Plugin Sandboxing is a major feature that was introduced in Deadline 8.0, however there is not a whole lot of information to be found about it. Part of the reason for this is that the Deadline Sandbox isn’t meant to be a user-facing feature – in fact, ideally our end users wouldn’t notice the switch at all.
Nonetheless, some of our astute users have noticed a series of mysterious “deadlinesandbox.exe” processes in their Task Manager, and it has generated some interest. Some details/limitations of its inner workings can also be relevant if you’ve been writing and maintaining a suite of custom scripts and plugins for Deadline, so I’d say it’s grand time to delve into the Sandbox with a bit more detail!
SO… WHAT IS THE DEADLINE SANDBOX?
The Sandbox itself is simply an executable that ships alongside the rest of our Deadline Client executables. It has no UI and it doesn’t do a whole lot on its own other than exit immediately; at first blush, it sure doesn’t sound very impressive. The Worker and Monitor applications know how to draw out its potential, however, and use it as a sub-process to run some of their python plugins.
More specifically, the Worker will create one sandbox for each render thread every time it dequeues a new Job. The general idea in this case is to give each new instance of our plugins free reign to modify its environment without having to worry about negatively affecting any future plugins, or the Worker itself.
Similarly, the Monitor, Worker, and Launcher will all use a dedicated Sandbox for running Event Plugin code. In an attempt to keep the number of processes down, all of the plugins are handled by a single sandbox (per application).
WHY COMPLICATE THINGS?
While segregation is generally frowned upon socially, and may seem completely unnecessary here at first glance, it is largely accepted as good practice from a computing perspective. The reasons are manifold, but the main benefit for Deadline is stability – which has been one of our big internal drivers for the Deadline 8.0 development cycle.
This improved stability is mainly achieved by the level separation between processes that the OS itself provides. It should now be much harder for a plugin (or its associated application) to affect the operation of the Worker itself.
Another benefit this provides is consistency. Since each plugin is run in a fresh Python process, it will no longer be affected by environment changes made several days ago, possibly in an entirely different plugin (nor will its own environment changes affect future plugins). Similarly, Python’s module caching issues should now also be a thing of the past; changes made to dependent modules should now be reflected much more transparently when imported (and without needing endless reload statements).
WHAT’S THE CATCH?
The obvious catch would be that if you have a suite of plugins that interact with each other and depend on environment changes to be persisted across runs. While we try to keep everything together at the Plugin level (pre/post Job/Task scripts all run in the same context as the Plugin itself), there will still be a disconnect after the current Job completes.
If you feel this separation of Plugins into their own processes introduces more headaches than solutions in your pipeline, the good news is that we largely anticipated this possibility. Your Workers can be configured to revert to the ‘old’ behavior, via a Repository-level setting that disables Plugin Sandboxing. You can find this option under the Worker Settings section of the Repository Configuration dialog in the Monitor (note you will need to restart any Workers that are currently running for them to pick up on this change).
AN EYE TOWARDS THE FUTURE
For us, the Sandbox represents even more than what was described above. It introduces all kinds of possibilities for further features; in fact, it’s already allowed us to make improvements to our Run-As-User functionality by running the entire plugin process as the Job User in 8.1. In the future, we might also look at tightening the reigns on the Sandbox a little, restricting the kind of things it might be able to do, in the name of security.
It also helps us pave the way towards having a single Worker process different Jobs on its own, instead of relying on running two separate Workers to maximize the utilization of each machine’s discrete resources.
We might also use this as a springboard to build up a suite of development tools that provide a dedicated environment to make Deadline Plugin and Script development more painless.
In the end, the Deadline Sandbox is so much more than a black box executable that seems to pollute your process list, it represents endless possibilities for future development. As always, feel free to let us know in which direction you’d like us to take it!