Deadline自製外掛筆記

用一個簡單的實作來當範例。
目的是:在 Houdini 對 Deadline 下算圖的時候,同時產生一個相依性 dependency 的 job,job 的內容是將算圖的輸出序列另外轉成 mp4 以方便預覽。

要達成這樣的目的,必須動用到 Deadline 的兩個外掛區塊:

  • Application Plugins 新增 FFmpeg 軟體外掛來轉檔。
  • Event Plugins 偵測有 job 下算的回調來新增 FFmpeg 轉檔工作。

新增軟體外掛 FFmpeg

其實 Deadline 就有內建 FFmpeg 的外掛,但對我來說有點麻煩,於是就改寫了一份。
外掛的詳細手冊

架構

自訂的腳本都放在 \DeadlineRepository\custom 裡面,當然你要放到原生資料夾去跟預設的外掛們擠位子也OK。

這邊要製作 FFmpeg 的外掛,於是就新增了資料夾 \DeadlineRepository\custom\plugins\FFmpeg\
資料夾新增好後,簡單版外掛通常會有四個檔案在裡面:.ico.param.options.py

除了 .py 是必須存在以外,其他三個檔案都是選擇性。

FFmpeg.ico

外掛的圖標

FFmpeg.param

外掛的環境設定檔,譬如說你可能需要指定 FFmpeg 的執行檔路徑,可以在這邊新增欄位設定。
而這檔案的內容都會跟 Deadline Monitor > Tools > Configure Plugins 即時連結。
如果你的 job 每個 task 都可以獨立算圖,那麼此檔案就是必備,要在內容設定 ConcurrentTasks 欄位並為 true

[ConcurrentTasks]
Type=label
Label=ConcurrentTasks
Category=About Plugin
CategoryOrder=-1
Index=0
Default=True

在這邊並沒有新增這個檔案。

FFmpeg.options

這個檔案是讓你在 job 按右鍵設定時,可以提供一個介面來方便修改外掛屬性,但 這邊的欄位submission的欄位下算的腳本(.py) 完全沒有自動關聯,需要手動去把上述地方的參數去跟下算的腳本做連結。
在此定義了 InputFileInputArgsOutputFileoutputArgs四個欄位,去對應到時 FFmpeg 的輸入與輸出。

[InputFile]
Type=filename
Label=Input File
Category=Input
Index=0
Required=false
DisableIfBlank=true

[InputArgs]
Type=string
Label=Input Arguments
Category=Input
Index=1
Required=false
DisableIfBlank=true

[OutputFile]
Type=filenamesave
Label=Output File
Category=Output
Index=0
Required=false
DisableIfBlank=true

[OutputArgs]
Type=string
Label=Output Arguments
Category=Output
Index=1
Required=false
DisableIfBlank=true

FFmpeg.py

最主要的就是這個下算腳本,先解釋一下 Deadline 下算的流程:

  • 藉由 軟體的 Deadline 外掛 或者 Deadline Monitor Submit指令 或者 其他手動指令 去新建 Job。
  • 建立 Job 的過程會產生 Job InfoPlugin Info 兩個檔案給 Deadline 總部。
  • Deadline 收到之後,便先根據 Job Info 的參數去進行佇列,等開始算圖時分發給算圖機。
  • 算圖機收到之後,再根據 Plugin Info 開啟對應的腳本,去執行工作。

所以這個 FFmpeg.py,便是取得 Job InfoPlugin Info 之後去執行算圖的東西。
詳細內容:

from System import *
from System.Diagnostics import *
from System.IO import *

from Deadline.Plugins import *
from Deadline.Scripting import *


# 下面兩個 function 是必備的,一個用來取得外掛物件,一個用來釋放記憶體
def GetDeadlinePlugin():
    return FFmpegPlugin()


def CleanupDeadlinePlugin(deadlinePlugin):
    deadlinePlugin.Cleanup()


# 外掛物件定義
class FFmpegPlugin(DeadlinePlugin):
    # 掛勾的回調
    def __init__(self):
        self.InitializeProcessCallback += self.InitializeProcess
        self.RenderExecutableCallback += self.RenderExecutable
        self.RenderArgumentCallback += self.RenderArgument

    # 釋放記憶體
    def Cleanup(self):
        for stdoutHandler in self.StdoutHandlers:
            del stdoutHandler.HandleCallback

        del self.InitializeProcessCallback
        del self.RenderExecutableCallback
        del self.RenderArgumentCallback
        del self.PreRenderTasksCallback
        del self.PostRenderTasksCallback

    # 起始程序
    def InitializeProcess(self):
        self.SingleFramesOnly = False

        # 錯誤直接導到 deadline
        self.StdoutHandling = True
        self.AddStdoutHandlerCallback(
            '.*Error.*').HandleCallback += self.HandleStdoutError

    # 算圖的程式路徑
    def RenderExecutable(self):
        # 如果電腦沒有 ffmpeg.exe,從網路硬碟複製一份到 我的文件夾 下
        user_path = Environment.GetFolderPath(
            Environment.SpecialFolder.MyDocuments)
        ffmpeg_path = user_path + '\\ffmpeg.exe'

        # 確認是否已有 ffmpeg.exe
        if File.Exists(ffmpeg_path):
            self.LogInfo('FFmpeg already installed.')
        else:
            self.LogInfo("FFmpeg didn't exists, Copying FFmpeg...")
            File.Copy('Q:\\DeadlinePlugin\\ffmpeg.exe', ffmpeg_path, True)

        return ffmpeg_path

    # 算圖的參數
    def RenderArgument(self):
        # 從 plugin info 取得輸入與輸出的參數
        inputFile = self.GetPluginInfoEntry('InputFile')
        inputArgs = self.GetPluginInfoEntry('InputArgs')
        outputFile = self.GetPluginInfoEntry('OutputFile')
        outputArgs = self.GetPluginInfoEntry('OutputArgs')

        # 輸出指令
        renderArgument = ''
        renderArgument += '%s ' % inputArgs
        renderArgument += '-i "%s" ' % inputFile
        renderArgument += '%s ' % outputArgs
        renderArgument += '-y "%s"' % outputFile

        return renderArgument

    def HandleStdoutError(self):
        self.FailRender(self.GetRegexMatch(0))

新增事件外掛 BeautyPreview

Deadline 可以根據各種事件回調,去執行自訂腳本
事件回調的手冊

架構

事件回調的腳本放在 \DeadlineRepository\custom\events 下,這裡創建了一個叫 BeautyPreview 的資料夾在裡面。
裡面必須要有兩個檔案:BeautyPreview.paramBeautyPreview.py

BeautyPreview.param

同軟體外掛一樣,是事件的環境設定,唯一的差別是這檔案在事件架構中是必須的,因為會關係到此事件外掛的開關。
設定是在 Deadline Monitor > Tools > Configure Events
這邊也只是簡單加上開關參數。

[State]
Type=Enum
Items=Global Enabled;Opt-In;Disabled
Label=State
Default=Disabled

BeautyPreview.py

這就是要執行的事件腳本了。

import re

from System.IO import *
from System.Text import *

from Deadline.Events import *
from Deadline.Scripting import *


# 跟軟體外掛一樣,有兩個必備的 function
def GetDeadlineEventListener():
    return BeautyPreviewEvent()


def CleanupDeadlineEventListener(deadlinePlugin):
    deadlinePlugin.Cleanup()


# 定義事件物件
class BeautyPreviewEvent(DeadlineEventListener):
    # 啟用的回調,這邊只有簡單一個:當工作 Submitted 後
    def __init__(self):
        self.OnJobSubmittedCallback += self.OnJobSubmitted

    # 釋放記憶體
    def Cleanup(self):
        del self.OnJobSubmittedCallback

    # 執行事件
    def OnJobSubmitted(self, job):
        # 如果這個 submit 的 job 不是 Houdini 或者 job 名稱不是 "previewit" 就略過
        if job.JobPlugin != "Houdini" or not job.JobName.endswith('previewit'):
            return
        self.LogInfo('> BeautyPreview: %s Triggered' % job.JobName)
        outputDirectories = job.JobOutputDirectories
        outputFilenames = job.JobOutputFileNames
        paddingRegex = re.compile("[^\\?#]*([\\?#]+).*")

        # 循環每個輸出路徑
        for i in range(0, len(outputFilenames)):
            # 取得輸出檔案路徑
            outputDirectory = outputDirectories[i]
            outputFilename = outputFilenames[i]
            outputPath = Path.Combine(outputDirectory, outputFilename)

            # 判斷檔名中的格數(####),轉換成 ffmpeg 看得懂的形式(%4d)
            m = re.match(paddingRegex, outputPath)
            if m is not None:
                padding = m.group(1)
                inputFilename = outputPath.replace(
                    padding, '%%%dd' % len(padding))
            else:
                inputFilename = outputPath

            # 輸出成 mp4 的路徑
            mp4Filename = Path.ChangeExtension(
                FrameUtils.GetFilenameWithoutPadding(outputPath), ".mp4")
            mp4Filename = mp4Filename.replace('#', '').strip(' .')

            # 記得前面說的流程嗎,這邊就是要創建 Job Info 跟 Plugin Info 兩個檔案來下算
            # Job Info 檔案
            jobInfoFilename = Path.Combine(
                ClientUtils.GetDeadlineTempPath(), "%s_preview_job_info.job" % job.JobId)
            writer = StreamWriter(jobInfoFilename, False, Encoding.Unicode)
            writer.WriteLine("Plugin=FFmpeg")
            writer.WriteLine("Name=%s" % (
                job.JobName.replace('previewit', '') + 'PREVIEW MP4'))
            writer.WriteLine("UserName=%s" % job.JobUserName)
            writer.WriteLine("Comment=Auto-Convert MP4 by BeautyPreview")
            writer.WriteLine("Pool=%s" % job.JobPool)
            writer.WriteLine("Priority=100")
            writer.WriteLine("MachineLimit=1")
            writer.WriteLine("Frames=0")
            writer.WriteLine("ChunkSize=1")
            writer.WriteLine("OutputFilename0=%s" % mp4Filename)
            writer.WriteLine("JobDependencies=%s" % job.JobId)  # 跟原本的 job 建立相依性
            writer.WriteLine("ResumeOnCompleteDependencies=true")  # 原本的 job 算完後啟動
            writer.Close()

            # Plugin Info 檔案
            pluginInfoFilename = Path.Combine(
                ClientUtils.GetDeadlineTempPath(), "%s_preview_plugin_info.job" % job.JobId)
            writer = StreamWriter(pluginInfoFilename, False, Encoding.Unicode)
            writer.WriteLine("InputFile=%s" % inputFilename)
            writer.WriteLine("InputArgs=-start_number %d -apply_trc iec61966_2_1" %
                             job.JobFramesList[0])  # 指定起始格數 跟 EXR轉檔的gamma設定
            writer.WriteLine("OutputFile=%s" % mp4Filename)
            writer.Close()

            # 下算
            ClientUtils.ExecuteCommand((jobInfoFilename, pluginInfoFilename))

額外補充 Submission

也可以幫軟體外掛寫了一個下算視窗,可以從 Deadline Monitor > Submit 或者 Deadline Lanucher > Submit執行。
將檔案放在 \DeadlineRepository\custom\scripts\Submission\FFmpegSubmission.py,注意有一個相似的資料夾是 \DeadlineRepository\custom\submission,不要搞錯。

這邊比較沒去做註解,因為是拿原本的改寫的

from System.Collections.Specialized import *
from System.IO import *
from System.Text import *

from Deadline.Scripting import *
from DeadlineUI.Controls.Scripting.DeadlineScriptDialog import DeadlineScriptDialog


# Globals
scriptDialog = None
settings = None


# Main Function Called By Deadline
def __main__():
    global scriptDialog
    global settings

    scriptDialog = DeadlineScriptDialog()
    scriptDialog.SetTitle("Submit FFmpeg Job To Deadline")
    scriptDialog.SetIcon(scriptDialog.GetIcon('FFmpeg'))

    scriptDialog.AddTabControl("Tabs", 0, 0)

    scriptDialog.AddTabPage("Job Options")
    scriptDialog.AddGrid()
    scriptDialog.AddControlToGrid(
        "Separator1", "SeparatorControl", "Job Description", 0, 0)
    scriptDialog.EndGrid()

    scriptDialog.AddGrid()
    scriptDialog.AddControlToGrid("NameLabel", "LabelControl", "Job Name", 0, 0,
                                  "The name of your job. This is optional, and if left blank, it will default to 'Untitled'.", False)
    scriptDialog.AddControlToGrid("NameBox", "TextControl", "Untitled", 0, 1)

    scriptDialog.AddControlToGrid("CommentLabel", "LabelControl", "Comment", 1, 0,
                                  "A simple description of your job. This is optional and can be left blank.", False)
    scriptDialog.AddControlToGrid("CommentBox", "TextControl", "", 1, 1)

    scriptDialog.EndGrid()

    scriptDialog.AddGrid()
    scriptDialog.AddControlToGrid(
        "Separator2", "SeparatorControl", "Job Options", 0, 0)
    scriptDialog.EndGrid()

    scriptDialog.AddGrid()
    scriptDialog.AddControlToGrid("PoolLabel", "LabelControl", "Pool",
                                  0, 0, "The pool that your job will be submitted to.", False)
    scriptDialog.AddControlToGrid("PoolBox", "PoolComboControl", "none", 0, 1)

    scriptDialog.AddControlToGrid("GroupLabel", "LabelControl", "Group",
                                  2, 0, "The group that your job will be submitted to.", False)
    scriptDialog.AddControlToGrid(
        "GroupBox", "GroupComboControl", "none", 2, 1)

    scriptDialog.AddControlToGrid("PriorityLabel", "LabelControl", "Priority", 3, 0,
                                  "A job can have a numeric priority ranging from 0 to 100, where 0 is the lowest priority and 100 is the highest priority.", False)
    scriptDialog.AddRangeControlToGrid("PriorityBox", "RangeControl", RepositoryUtils.GetMaximumPriority(
    ) / 2, 0, RepositoryUtils.GetMaximumPriority(), 0, 1, 3, 1)

    scriptDialog.AddControlToGrid("TaskTimeoutLabel", "LabelControl", "Task Timeout", 4, 0,
                                  "The number of minutes a slave has to render a task for this job before it requeues it. Specify 0 for no limit.", False)
    scriptDialog.AddRangeControlToGrid(
        "TaskTimeoutBox", "RangeControl", 0, 0, 1000000, 0, 1, 4, 1)
    scriptDialog.AddSelectionControlToGrid("AutoTimeoutBox", "CheckBoxControl", False, "Enable Auto Task Timeout", 4, 2,
                                           "If the Auto Task Timeout is properly configured in the Repository Options, then enabling this will allow a task timeout to be automatically calculated based on the render times of previous frames for the job. ", False)

    scriptDialog.AddControlToGrid("MachineLimitLabel", "LabelControl", "Machine Limit", 6, 0,
                                  "Use the Machine Limit to specify the maximum number of machines that can render your job at one time. Specify 0 for no limit.", False)
    scriptDialog.AddRangeControlToGrid(
        "MachineLimitBox", "RangeControl", 0, 0, 1000000, 0, 1, 6, 1)
    scriptDialog.AddSelectionControlToGrid("IsBlacklistBox", "CheckBoxControl", False, "Machine List Is A Blacklist", 6, 2,
                                           "You can force the job to render on specific machines by using a whitelist, or you can avoid specific machines by using a blacklist.")

    scriptDialog.AddControlToGrid("MachineListLabel", "LabelControl", "Machine List",
                                  7, 0, "The whitelisted or blacklisted list of machines.", False)
    scriptDialog.AddControlToGrid(
        "MachineListBox", "MachineListControl", "", 7, 1, colSpan=2)

    scriptDialog.AddControlToGrid("LimitGroupLabel", "LabelControl",
                                  "Limits", 8, 0, "The Limits that your job requires.", False)
    scriptDialog.AddControlToGrid(
        "LimitGroupBox", "LimitGroupControl", "", 8, 1, colSpan=2)

    scriptDialog.AddControlToGrid("DependencyLabel", "LabelControl", "Dependencies", 9, 0,
                                  "Specify existing jobs that this job will be dependent on. This job will not start until the specified dependencies finish rendering.", False)
    scriptDialog.AddControlToGrid(
        "DependencyBox", "DependencyControl", "", 9, 1, colSpan=2)

    scriptDialog.AddControlToGrid("OnJobCompleteLabel", "LabelControl", "On Job Complete", 10,
                                  0, "If desired, you can automatically archive or delete the job when it completes.", False)
    scriptDialog.AddControlToGrid(
        "OnJobCompleteBox", "OnJobCompleteControl", "Nothing", 10, 1)
    scriptDialog.AddSelectionControlToGrid("SubmitSuspendedBox", "CheckBoxControl", False, "Submit Job As Suspended", 10, 2,
                                           "If enabled, the job will submit in the suspended state. This is useful if you don't want the job to start rendering right away. Just resume it from the Monitor when you want it to render.")
    scriptDialog.EndGrid()

    scriptDialog.AddGrid()
    scriptDialog.AddControlToGrid(
        "Separator3", "SeparatorControl", "FFmpeg Options", 0, 0, colSpan=2)

    scriptDialog.AddControlToGrid(
        "InputLabel", "LabelControl", "Input File", 1, 0, "The input file to process.", False)
    scriptDialog.AddSelectionControlToGrid("InputBox", "FileBrowserControl", "",
                                           "AVI Files (*.avi);;M2V Files (*.m2v);;MPG Files (*.mpg);;VOB Files (*.vob);;WAV Files (*.wav);;All Files (*)", 1, 1)

    scriptDialog.AddControlToGrid("InputArgsLabel", "LabelControl", "Input Arguments",
                                  2, 0, "Additional command line arguments for the input file. ", False)
    scriptDialog.AddControlToGrid("InputArgsBox", "TextControl", "", 2, 1)

    scriptDialog.AddControlToGrid(
        "OutputLabel", "LabelControl", "Output File", 4, 0, "The output file path.", False)
    scriptDialog.AddSelectionControlToGrid(
        "OutputBox", "FileSaverControl", "", "All Files (*)", 4, 1)

    scriptDialog.AddControlToGrid("OutputArgsLabel", "LabelControl", "Output Arguments",
                                  5, 0, "Additional command line arguments for the output file. ", False)
    scriptDialog.AddControlToGrid("OutputArgsBox", "TextControl", "", 5, 1)

    scriptDialog.EndGrid()
    scriptDialog.EndTabPage()

    scriptDialog.EndTabControl()

    scriptDialog.AddGrid()
    scriptDialog.AddHorizontalSpacerToGrid("HSpacer1", 0, 0)
    submitButton = scriptDialog.AddControlToGrid(
        "SubmitButton", "ButtonControl", "Submit", 0, 1, expand=False)
    submitButton.ValueModified.connect(SubmitButtonPressed)
    closeButton = scriptDialog.AddControlToGrid(
        "CloseButton", "ButtonControl", "Close", 0, 2, expand=False)
    closeButton.ValueModified.connect(scriptDialog.closeEvent)
    scriptDialog.EndGrid()

    # Application Box must be listed before version box or else the application changed event will change the version
    settings = ("CategoryBox", "PoolBox", "GroupBox", "PriorityBox",
                "MachineLimitBox", "IsBlacklistBox", "MachineListBox", "LimitGroupBox", "SceneBox", "FramesBox", "ChunkSizeBox")
    scriptDialog.LoadSettings(GetSettingsFilename(), settings)
    scriptDialog.EnabledStickySaving(settings, GetSettingsFilename())

    scriptDialog.ShowDialog(False)


def GetSettingsFilename():
    return Path.Combine(ClientUtils.GetUsersSettingsDirectory(), "FFmpegSettings.ini")


def SubmitButtonPressed(*args):
    global scriptDialog

    # Check if input file exist.
    inputFile = scriptDialog.GetValue("InputBox")

    if(inputFile == ""):
        scriptDialog.ShowMessageBox("No input file specified", "Error")
        return
    elif (PathUtils.IsPathLocal(inputFile)):
        result = scriptDialog.ShowMessageBox(
            "Input file %s is local.  Are you sure you want to continue?" % inputFile, "Warning", ("Yes", "No"))
        if(result == "No"):
            return

    # Check output file.
    outputFile = (scriptDialog.GetValue("OutputBox")).strip()

    if(outputFile == ""):
        scriptDialog.ShowMessageBox("No output file specified", "Error")
        return
    else:
        if(PathUtils.IsPathLocal(outputFile)):
            result = scriptDialog.ShowMessageBox(
                "Output file %s is local, are you sure you want to continue?" % outputFile, "Warning", ("Yes", "No"))
            if(result == "No"):
                return

    # Submit each scene file separately.
    # Create job info file.
    jobInfoFilename = Path.Combine(
        ClientUtils.GetDeadlineTempPath(), "ffmpeg_job_info.job")
    writer = StreamWriter(jobInfoFilename, False, Encoding.Unicode)
    writer.WriteLine("Plugin=FFmpeg")
    writer.WriteLine("Name=%s" % scriptDialog.GetValue("NameBox"))
    writer.WriteLine("Comment=%s" % scriptDialog.GetValue("CommentBox"))
    writer.WriteLine("Pool=%s" % scriptDialog.GetValue("PoolBox"))
    writer.WriteLine("Group=%s" % scriptDialog.GetValue("GroupBox"))
    writer.WriteLine("Priority=%s" % scriptDialog.GetValue("PriorityBox"))
    writer.WriteLine("TaskTimeoutMinutes=%s" %
                     scriptDialog.GetValue("TaskTimeoutBox"))
    writer.WriteLine("EnableAutoTimeout=%s" %
                     scriptDialog.GetValue("AutoTimeoutBox"))

    writer.WriteLine("MachineLimit=%s" %
                     scriptDialog.GetValue("MachineLimitBox"))
    if(bool(scriptDialog.GetValue("IsBlacklistBox"))):
        writer.WriteLine("Blacklist=%s" %
                         scriptDialog.GetValue("MachineListBox"))
    else:
        writer.WriteLine("Whitelist=%s" %
                         scriptDialog.GetValue("MachineListBox"))

    writer.WriteLine("LimitGroups=%s" % scriptDialog.GetValue("LimitGroupBox"))
    writer.WriteLine("JobDependencies=%s" %
                     scriptDialog.GetValue("DependencyBox"))
    writer.WriteLine("OnJobComplete=%s" %
                     scriptDialog.GetValue("OnJobCompleteBox"))

    if(bool(scriptDialog.GetValue("SubmitSuspendedBox"))):
        writer.WriteLine("InitialStatus=Suspended")

    writer.WriteLine("Frames=0")
    writer.WriteLine("ChunkSize=1")
    writer.WriteLine("OutputFilename0=%s" % outputFile)

    writer.Close()

    # Create plugin info file.
    pluginInfoFilename = Path.Combine(
        ClientUtils.GetDeadlineTempPath(), "ffmpeg_plugin_info.job")
    writer = StreamWriter(pluginInfoFilename, False, Encoding.Unicode)

    writer.WriteLine("InputFile=%s" % scriptDialog.GetValue("InputBox").strip())
    writer.WriteLine("InputArgs=%s" % scriptDialog.GetValue("InputArgsBox").strip())

    writer.WriteLine("OutputFile=%s" % outputFile)
    writer.WriteLine("OutputArgs=%s" % scriptDialog.GetValue("OutputArgsBox").strip())

    writer.Close()

    # Setup the command line arguments.
    arguments = StringCollection()

    arguments.Add(jobInfoFilename)
    arguments.Add(pluginInfoFilename)

    # Now submit the job.
    results = ClientUtils.ExecuteCommandAndGetOutput(arguments)
    scriptDialog.ShowMessageBox(results, "Submission Results")