Deadline Monitor

• 1 Deadline Monitor
  • 1.1 Overview
  • 1.2 Running The Monitor
  • 1.3 Job Panel
    • 1.3.1 Job Right-Click Options
  • 1.4 Task Panel
    • 1.4.1 Task Right-Click Options
  • 1.5 Slave/Pulse Panel
    • 1.5.1 Slave List
    • 1.5.2 Slave Right-Click Options
    • 1.5.3 Pulse List
    • 1.5.4 Pulse Right-Click Options
  • 1.6 Limit Panel
  • 1.7 Monitor Menus
    • 1.7.1 File Menu
    • 1.7.2 View Menu
    • 1.7.3 Submit Menu
    • 1.7.4 Scripts Menu
    • 1.7.5 Tools Menu
    • 1.7.6 Help Menu
  • 1.8 Administrative Options
  • 1.9 FAQ

Overview

The Deadline Monitor application offers detailed information and control options for each job and slave in your Deadline Repository. It provides normal users a means of monitoring and controlling their jobs, and it gives administrators options for configuring and controlling the entire render farm

The Monitor contains four main panels. The Job Panel shows all the jobs that are currently in the queue and the Task Panel shows all tasks and additional information for the selected job. The Slave/Pulse panel provides information about all the machines in the render farm, and the Limit Panel shows the available Limits and their usage. You can customize the look and feel of the Monitor by referring to the Monitor Customization documentation.

The Monitor allows you to access several options, which are split into general and administrative options. Administrative options require the user to be in Super User mode, which is done by selecting Tools -> Super User Mode. Super User mode can be password protected simply by specifying a password in the User Security section of the Repository Options. It is also possible to remote into slave machines directly from the Monitor using VNC, RDC, ARd, or Radmin. More details can be found in the Remote Access documentation.

Running The Monitor

To start the Monitor:

• On Windows, you can start the Monitor from the Start Menu, from the Deadline Launcher, or from a command prompt.
• On Linux, you can start the Monitor from the Deadline Launcher or from a terminal window.
• On Mac OSX, you can start the Monitor by double-clicking the Monitor application in Finder, from the Deadline Launcher, or from a terminal window.

Command line info:

• When running the Monitor from a command prompt or terminal window, this is the command to use:

deadlinelauncher -monitor

• To start the Monitor without a splash screen, you can run the following:

deadlinemonitor -nosplash

• To allow for more than one Monitor to run at a time, you can run the following:

deadlinemonitor -multi

• To shutdown the Monitor if it's already running, you can run the following:

deadlinemonitor -shutdown

If you're launching the Monitor for the first time on your machine, you will be prompted with a Login dialog. Simply choose your user name or create a new one before continuing.

Job Panel

The job panel contains a list that shows all the jobs that have been submitted to the Deadline Repository. It also displays useful information about each job such as it's name, user, status, error count, plug-in, etc. You can customize the columns in the job list from the View menu, and you can use the Job Filter toolbar to filter out jobs based on certain properties. Finally, you can use the search box to find specific jobs in the remaining jobs that are listed.

Jobs can be in one of many states, which are explained in detail in the How Deadline Works documentation. As the jobs enter different states, their color changes. When jobs are active, they first appear as green in the list. A job will remain green as it continues to render, but if it starts to accumulate errors, it will start to turn brown and eventually red. This allows you to see at a glance which jobs are having problems. For more information on job monitoring, see the Job Monitoring documentation.

You can refresh a job in the list by clicking or double-clicking on it. You can also access many options for a job, which are listed below, by right-clicking on it. Note that the availability of these options can vary depending on the context in which they are used.

Job Right-Click Options

Refresh Job

Refreshes the selected job.

Refresh All Jobs

Refreshes all the jobs in the Monitor, and loads any new jobs that have been submitted since the last Monitor refresh.

Load New Jobs

Loads new jobs that have been submitted since the last Monitor refresh, without refreshing the jobs that are already displayed.

Resume Failed Job

Resumes a job that is in the failed state. When you resume the job, you have the option to ignore Failed Job Detection or Failed Task Detection in the future. Note that this option is only visible when the jobs selected are in the failed state.

Release Pending Job

Releases a pending job, and removes all dependency and scheduling settings.

Make All Tasks Pending

For frame dependent jobs, moves all non-pending tasks to the pending state.

Resume Job

Resumes a job that is in the suspended state.

Suspend Job

Suspends a job that is in the queued or active state. If you suspend a job that is in the active state, any tasks that are currently rendering will be requeued.

Requeue Job

Requeues all the tasks for a job that is in the queued, active, suspended, or completed state.

Delete Job

Deletes a job. The job can be in any state.

Archive Job

Archives a job that is suspended, completed, or failed.

Un-Archive Job

Un-archives an archived job.

Resubmit Job

Allows you to resubmit an existing job with a new frame range and group size. If you want to tweak some settings before the job renders, you can enable the option to suspend the job on submission.

Explore Output

View the directory where a job's output is sent. If a job has multiple output directories, they will be listed in a separate submenu. This option to view the output directory is available if the job's output directory was specified by the script that submitted the job.

Copy Output Path

Copies a job's output directory to the clipboard. If a job has multiple output directories, they will be listed in a separate sub-menu. This option to view the output directory is available if the job's output directory was specified by the script that submitted the job.

Scan For Missing Output

Scans a job's output directory for any tasks that have missing output, or output with a size less than or equal to the size specified. If any are found, you are given the option to requeue those tasks (simply place a check mark beside the tasks to requeue). This option is available if Deadline is aware of a job's output filename, which is specified by the script that submitted the job.

Add To Job Monitor

Adds the selected job to the Job Monitor application. If the Job Monitor isn't already running, it will be launched with the selected job already added.

Retrieve Data File

Save the scene file that was submitted with a job to another location. This option is available if a scene file was submitted with a job as an auxiliary file.

Explore Repository Directory

Browse a job's directory in the Repository.

Notify Me When Complete

If you select this option, you will be notified when the selected job completes.

Modify Notification Users

Modify which users will receive a notification when this job completes.

Modify Notification Note

Add a note which is appended to the notification that is sent to the user(s) when this job finishes.

Email User

Send a direct email to a job's user using your default mail application.

Net Send User

Send a message to a job's user. The user must have the Deadline Launcher running on their machine.

Growl Notify User

Send a Growl notification to a job's user. The user must have Growl installed on their machine.

Modify Properties / Machine Limit

Modify the properties or the machine limit of a job. See the Modifying Job Properties documentation for more information.

Modify Dependencies

You can set which jobs that this job is dependent on. See the Modifying Job Properties documentation for more information.

Modify Frame Range

Allows you to modify the frame range and group size of a job. See the Modifying Job Properties documentation for more information.

Modify Scheduling

This option allows you to schedule a job to run at a specific time once, or on daily intervals. See the Modifying Job Properties documentation for more information.

View Dependencies

Allows you to view the dependencies of a job. This option is available if the job is dependent on one or more jobs.

View Bad Slaves List

Allows you to view the slaves that have been marked bad for this job. See the Modifying Job Properties documentation for more information.

Scripts

Launch job scripts from here. You can add your own custom python job scripts to the Scripts menu. See the Monitor Scripting documentation for more details.

View History

View the entire history of a job. This includes any modifications to the job's properties, if and when it was ever suspended, etc.

View Errors

View the errors for a job. See the Handling Job Errors section of the Job Monitoring documentation for more information.

View Logs

View the task logs for a job. All logs for successful, failed, and requeued tasks can be viewed here.

View Requeues

View the task requeues for a job. Selecting a requeue report will show why the task was requeued. The rendering time wasted due to requeue is shown in the top-right corner.

Task Panel

The task panel shows all the tasks for the job that is currently selected. It displays useful information about each task such as its frame list, status, and error count, and you can customize the columns in the task list from the View menu. You can also see additional information for the selected job, as well as a graph of the task render times, by clicking on the appropriate tabs.

There is also a Task Statistics panel that you can toggle on and off from the View -> Layout menu. This panel will display histograms showing how the CPU and memory usage of the selected task changed during the course of the render.

You can remote into the machine that is currently rendering a task by double-clicking on it. If the task has finished, you can open its output by also double-clicking on it. Finally, you can access many options for a task, which are listed below, by right-clicking on it. Note that the availability of these options can vary depending on the context in which they are used.

Task Right-Click Options

Refresh Job

Refresh the job that is currently selected in the job list.

Connect by VNC/ARD/Remote Desktop/Radmin

Connect to the machine that rendered or is rendering the task. These options are only available if their corresponding settings in the Repository Options allow for remote connections.

Send Net Message/Growl Notification

Send a net message or Growl notification to the machine that rendered or is rendering the task.

[output file name] - Default Viewer

View the output for this task. If the task contains multiple frames, an image viewer dialog will appear, allowing you to choose which output to view. This option is available if Deadline is aware of a job's output filename, which is specified by the script that submitted the job.

Copy Render Path - [output file path]

Copy the output file path for this task to the clipboard. This option is available if Deadline is aware of a job's output filename, which is specified by the script that submitted the job.

Requeue Task

Requeues a completed task. This option is available if the task is in the completed state only.

Mark Task As Complete

Sets the task to the completed state. This option is available if the task is in the queued, rendering, suspended, or failed state.

Release Pending Task

If the job is frame dependent, releases a pending task. This option is available if the task is in the pending state only.

Mark Task As Pending

If the job is frame dependent, sets the task to the pending state. This option is available if the task is in the queued, rendering, suspended, or failed, or completed state.

Mark Task As Failed

Sets the task to the failed state. This option is available if the task is in the queued, rendering, suspended, or completed state.

Resume Failed Task

Resumes a failed task. If the entire job is failed, it will re-enter the queued state. This option is available if the task is in the failed state only.

Requeue Tasks by Slave

Requeues all the completed tasks rendered by this particular slave. This can be useful if a particular slave is constantly rendering bad frames. This option is available if there is at least one task in the completed state for that particular slave.

Blacklist/Un-Whitelist Slave

If the job has a blacklist, add the slave that rendered or is rendering the task to it. If the job has a whitelist, remove the slave from it. This option is available if the task is rendering or completed.

Find Slave In Slave List

Selects the slave in the slave list that has rendered or is currently rendering the selected task. This option is only available if the Slave List is visible, which can be toggled from the View menu.

Submit Tasks As New Job

Submits the selected tasks as a new job. The new job will have the exact same properties as the original job. Note that the output produced by these tasks for the new job will overwrite output from the existing job, since the same output settings will be used. You can choose to submit the job in the suspended state so that you can tweak its properties before resuming it.

Task Reports

View the errors, logs, and requeue reports for a specific task.

Slave/Pulse Panel

This panel contains separate tabs for the slave and pulse lists.

Slave List

This slave list shows all the slaves that are in your Deadline render farm. It shows system information about each slave, as well as information about the job the slave is currently rendering. You can customize the columns in the slave list from the View menu (while the slave list is visible), and you can use the Slave Filter toolbar to filter out slaves based on certain properties. Finally, you can use the search box to find specific slaves in the remaining slaves that are listed.

There is also a Slave Availability filter in the Slave Filter toolbar. If you click on a job with this filter enabled, the Slave list will be filtered to only show the slaves that can render the selected job. This takes into account the Pool, Group, White/Blacklisted slaves, and any Limits the job uses. This makes it easy to figure out why a job might not be rendering when you think it should be.

If you want the slave list to show which machines are online, you can select View -> Ping Slave. This will show if the machine can be pinged under the Ping column. Note that when this is enabled, the speed at which the slave list refreshes will likely be affected.

If you see a slave that is colored orange in the list, this means that the slave is using up 95% or more of its available RAM. This is normally not a problem, and is just a means of showing which machines are approaching their memory limit. If you see a slave that is colored cyan in the list, this means that the slave is using a license file that will expire shortly, or already has expired.

You can remote into a slave by double-clicking on it. You can also access many options for a slave, which are listed below, by right-clicking on it. Note that the availability of these options can vary depending on the context in which they are used (for example, you may need to be in Super User Mode).

Slave Right-Click Options

Refresh Slave

Refreshes the selected slave.

Refresh All Slaves

Refreshes the slave list.

Connect by VNC/ARD/Radmin/Remote Desktop

Remote into the selected slave(s) via the selected method. These options are only available if their corresponding settings in the Repository Options allow for remote connections.

Ping Slave

Pings the machine to see if it is online. The results will show up in the Ping column in the Slave list.

Send Net Message/Growl Notification

Send a net message or Growl notification to the selected slave(s).

Modify Slave Settings

Change the settings for the slave. See the Slave Settings documentation for more information.

Modify Pools/Groups

Assign slaves to pools and groups. See the Pools and Groups documentation for more information.

Find Task In Task List

Selects the task in the task list that this slave is currently rendering. This option is only available if the slave is currently rendering a task.

Enabled/Disabled

Enable or disable a slave. The Slave application will not start up on slaves that are disabled.

Mark Slave As Offline

Mark a stalled slave as being offline. This is useful if a slave didn't close cleanly when it was last shutdown, and you don't plan on restarting it for a while. This option is available if the slave is stalled.

Delete Slave

Delete a slave from your Deadline render farm. Deleting a slave from the Monitor does not uninstall the Deadline Client applications on the slave, it simply removes Deadline's knowledge of that slave's existence. To add the slave back again, simply start the Slave application on that machine.

Remote Control

Allows you to control the Slave application or the machine that it is running on remotely. See the Remote Control and Access documentation for more information.

Scripts

Launch slave scripts from here. You can add your own custom python slave scripts to the Scripts menu. See the Monitor Scripting documentation for more information.

View History

View the entire history of a slave. This includes when the slave was launched, when the slave was VNC'ed into, etc.

View Errors

View the errors for a slave.

Clear Errors

Remove all the errors for a slave. This option is available if the slave has reported any errors.

Pulse List

The pulse list shows which machine Pulse is running on. It also displays useful information about the machine such as its name, listening port, status, etc, as well as some useful memory and CPU information. You can customize the columns in the pulse list from View menu (while the pulse list is visible).

If you have temperature sensors configured in Power Management, and you want to see their current temperatures, you can hover the mouse over the Zone Temperatures label in the upper right-hand corner of the panel.

You can remote into a machine by double-clicking on it. You can also access many options, which are listed below, by right-clicking on it. Note that the availability of these options can vary depending on the context in which they are used (for example, you may need to be in Super User Mode).

Pulse Right-Click Options

Refresh List

Refreshes the Pulse list.

Connect by VNC/ARD/Radmin/Remote Desktop

Remote into the selected machine(s) via the selected method. These options are only available if their corresponding settings in the Repository Options allow for remote connections.

Send Net Message/Growl Notification

Send a net message or Growl notification to the selected machine(s).

Remote Control

Allows you to control the Pulse application or the machine that it is running on remotely.

When executing an arbitrary command, if you want to execute a DOS command on a Windows machine, the command must be preceded with "cmd /C". This opens the DOS prompt, executes the command, and closes the prompt. For example:

cmd /C echo "foo" > c:\test.txt

View History

Allows you to view the entire history of Pulse. This includes when the Pulse was launched, when Pulse was VNC'ed into, etc. This option is available if you are in super user mode.

Auto Configure Pulse

Use this to automatically set the selected machine as the Pulse server for your repository. You can choose to use the machine's host name or IP address in the configuration. To set the Pulse server settings manually, refer to the Pulse Settings section of the Repository Options.

Mark Pulse As Offline

Allows you to mark a stalled pulse as being offline. This is useful if pulse didn't close cleanly when it was last shutdown, and you don't plan on restarting it for a while. This option is available if the pulse is stalled.

Delete Pulse

Allows you to delete Pulse from your Deadline render farm. Deleting Pulse from the Monitor does not uninstall the Deadline Client applications on the machine, it simply removes Deadline's knowledge of Pulse's existence. To add Pulse back again, simply start the Pulse application on that machine.

Limit Panel

The limit group panel contains a list that shows all the Limits that are in your Deadline repository. It also displays useful information about each Limit such as its name, limit, in use count, and blacklisted slaves. You can customize the columns in the job list from the View menu. You can access many options for the Limits by right-clicking on them. See the Limits documentation for more information.

Monitor Menus

File Menu

Change Repository

Connect to a different repository, or reconnect to the current repository if the Monitor becomes disconnected.

Change User

Change the current Deadline user. You have the choice to select a different user or create a new one.

Satellite Mode

Enables Satellite Mode. See the Satellite Mode documentation for more information.

View Menu

Refresh

Refresh the entire Monitor, or individual lists. Select a refresh option from the drop down in the Main Toolbar to choose which option is performed when you use the Refresh shortcut.

Toolbars

Toggle the visibility of the Main, Job, and Slave toolbars.

Layout

Change the horizontal layout of the Job and Task panels to a vertical one, and vice versa. You can also toggle the visibility of the Slave/Pulse and Limit panels.

Columns

Configure the order and visibility of the columns for each list.

Ego Centric Sort

If enabled, the current user's jobs will always be displayed at the top of the jobs list. This makes it easy for a user to find their jobs quickly.

Ping Slave Machines

If enabled, the slave machines will be pinged during a refresh to check if they are online. This affects the speed at which the slave list is refreshed, which is why we've included the option to enable or disable it. If disabled, you will see Disabled displayed under the Ping column in the slave list.

Keyboard Shortcuts

Enable or disable the use of keyboard shortcuts. When disabled, the shortcuts will not appear in the menus.

Grid Lines

Display or hide the grid lines in the lists.

Submit Menu

Submit jobs to Deadline for different rendering plug-ins. More information regarding the Monitor submission scripts for each plug-in can be found in the Deadline Plug-ins documentation. You can also add your own submission dialogs to the submission menu. Check out the Monitor Scripting documentation for more information.

Scripts Menu

Run custom scripts that you can create. Check out the Monitor Scripting documentation for more information.

Tools Menu

Super User

Enter Super User Mode, which allows you to access the administrative Monitor options. Super User mode can be password protected simply by specifying a password in the Access Control section of the Repository Options.

Launch Job Monitor

Launches the Job Monitor application, which allows you to monitor the progress of specific jobs without having to refresh the Monitor.

Options

Modify the Monitor options and user settings. The Monitor options are described in the Monitor Customization documentation, and the user settings are described in the User Management documentation.

Edit Themes

Create new and edit existing themes for the Monitor and other Deadline applications. More information can be found in the Monitor Customization documentation.

Help Menu

Online Manual

Opens the online manual for Deadline in your default browser.

Technical Support

Opens the online technical support page in your default browser.

Administrative Options

Administrative options can be accessed from the Tools menu, and require the Monitor to be in Super User mode, which is done by selecting Tools -> Super User Mode. Super User mode can be password protected simply by specifying a password in the Access Control section of the Repository Options.

View History

View the repository's history. This includes when jobs were submitted, when settings were modified, etc.

View Power Management History

View the Power Management history. This includes when and why slaves were started up and shutdown.

View Repository Statistics

View the statistics that have been collected. See the Repository Statistics documentation for more information.

Manage Users

Add and remove users from the repository and change their user settings. See the User Management documentation for more information.

Manage Pools

Add and remove pools from the repository, and assign them to slaves. See the Pools and Groups documentation for more information on pools and how they affect job scheduling.

Manage Groups

Add and remove groups from the repository, and assign them to slaves. See the Pools and Groups documentation for more information on groups and how they affect job scheduling.

Explore Repository Root

Explore the Deadline Repository directory.

Cleanup Completed Jobs

Automatically delete or archive jobs that completed prior to a specified date.

Cleanup Archived Completed Jobs

Automatically delete archived jobs that completed prior to a specified date.

Undelete Jobs

Use this as a last ditch attempt to restore jobs that have been deleted (there is a one hour grace period before the job is removed permanently). Note that once a job is deleted, its error and log reports cannot be recovered with this tool.

Check For Orphaned Limit Stubs

When a Slave requires a Limit for a job, it obtains a Limit Stub which adds to the Limit's usage count. If the Slave exits uncleanly, it's possible that the Stub can become orphaned and continue to count against the Limit's total usage. Deadline will eventually clean this up on its own, but this option can be used to clean things up immediately if you suspect there is a problem.

Check For Stalled Slaves

When a Slave hasn't updated its state for a long time, Deadline will eventually detect this and mark the Slave as stalled (stalled Slaves appear red in the Slave list). This option can be used to check for stalled Slaves immediately if you suspect there is a problem.

Empty Trash Bin

Empties the trash folder in the repository. Deadline will eventually clean this up on its own, but this option can be used to clean things up immediately if you wish.

Suspend All Active Jobs

This acts like a big Pause button that suspends all queued, active, and pending jobs in the queue. The list of the jobs that are suspended using this command are stored so that they can later be resumed by using the Resume Previously Suspended Jobs option.

Resume Previously Suspended Jobs

Resumes all the jobs that were suspended using the Suspend All Active Jobs command. Suspended jobs that were not suspended by the Suspend All Active Jobs command will not be affected.

Configure Repository Options

Configure the global repository options. See the Repository Options documentation for more information.

Configure Plugins

Configure the Deadline plug-ins. See the Deadline Plug-ins documentation for details about the configuration for each plug-in that is shipped with Deadline.

Configure Event Plugins

Configure the Deadline event plug-ins. See the Event Plug-in Scripting documentation for more information on creating Event Plug-ins.

Configure Power Management

Configure the Power Management settings. See the Power Management documentation for more information.

FAQ

Why are all the Remote Control options in the Deadline Monitor are greyed out?

In order to use the Remote Control options, you have to enable Remote Administration in the Repository Options.

Is it possible to have the Monitor refresh automatically?

Yes, this can be set in the user options (Tools -> Options). Under Monitor options, find the User Interface section and set the Auto Refresh Period.

When I try to remote desktop to Slaves using the Monitor on Windows XP64, I get an error that says "Cannot find remote desktop connection application".

The problem is likely that Deadline doesn't have the required permissions to read or execute the files in the C:\WINDOWS\system32 directory, where the RDC executable (mstsc.exe) is located. This seems to be built into .NET, so I'm not sure if we can get around it. You might have to play around with the user/security settings to allow Deadline access to the RDC executable file.

What does the Purge Obsolete Pools checkbox in the Manage Pools dialog do?

This purges any pools that have been removed that are NOT assigned to any slaves or jobs. If either a single job or slave is still assigned that pool, it won't be purged (doing so could cause some unexpected results).

What does it mean when a slave is orange?

This means that the slave is using up 95% or more of its available RAM. This is normally not a problem, and is just a means of showing which machines are approaching their memory limit.