:index:`DAGMan Quick Reference<single: DAGMan; DAGMan Quick Reference>`

Quick Reference
===============

DAG Commands
------------

General
^^^^^^^

:dag-cmd-def:`INCLUDE` (see :ref:`Full Description<DAG Include cmd>`)
    Parse the provided file as if it was inline to the current file.

    .. code-block:: condor-dagman

        INCLUDE filename

:dag-cmd-def:`JOB` (see :ref:`Full Description<DAGMan JOB>`)
    Create a normal DAG node to execute specified HTCondor jobs.

    .. code-block:: condor-dagman

        JOB NodeName SubmitDescription [DIR directory] [NOOP] [DONE]

:dag-cmd-def:`PARENT/CHILD` (see :ref:`Full Description<DAG node dependencies>`)
    Create dependencies between two or more DAG nodes.

    .. code-block:: condor-dagman

        PARENT ParentNodeName [ParentNodeName2 ... ] CHILD  ChildNodeName [ChildNodeName2 ... ]

:dag-cmd-def:`SPLICE` (see :ref:`Full Description<DAG splicing>`)
    Incorporate the specified DAG file into the structure of another DAG.

    .. code-block:: condor-dagman

        SPLICE SpliceName DagFileName [DIR directory]

:dag-cmd-def:`SUBDAG` (see :ref:`Full Description<subdag-external>`)
    Specify a DAG workflow to be submitted by :tool:`condor_submit_dag` and managed
    by a parent DAG.

    .. code-block:: condor-dagman

        SUBDAG EXTERNAL JobName DagFileName [DIR directory] [NOOP] [DONE]

:dag-cmd-def:`SUBMIT-DESCRIPTION` (see :ref:`Full Description<DAG submit description cmd>`)
    Create an inline job submit description that can be applied to multiple
    DAG nodes.

    .. code-block:: condor-dagman

        SUBMIT-DESCRIPTION DescriptionName {
            # submit attributes go here
        }

Node Behavior
^^^^^^^^^^^^^

:dag-cmd-def:`DONE`
    Mark a DAG node as done causing neither the associated jobs or scripts to execute.

    .. code-block:: condor-dagman

        DONE NodeName

:dag-cmd-def:`PRE_SKIP` (see :ref:`Full Description<Node pre skip cmd>`)
    Inform DAGMan to skip the remaining node execution if that nodes specified PRE
    script exits with a specified code.

    .. code-block:: condor-dagman

        PRE_SKIP <NodeName | ALL_NODES> non-zero-exit-code

:dag-cmd-def:`PRIORITY` (see :ref:`Full Description<DAG Node Priorities>`)
    Assign a node priority to control DAGMan node submission.

    .. code-block:: condor-dagman

        PRIORITY <NodeName | ALL_NODES> PriorityValue

:dag-cmd-def:`RETRY` (see :ref:`Full Description<Retry DAG Nodes>`)
    Inform DAGMan to retry a node up to a specified number of times when a failure
    occurs.

    .. code-block:: condor-dagman

        RETRY <NodeName | ALL_NODES> NumberOfRetries [UNLESS-EXIT value]

:dag-cmd-def:`SCRIPT` (see :ref:`Full Description<DAG Node Scripts>`)
    Apply a script to be executed on the AP for a specified node.

    .. code-block:: condor-dagman

        # PRE-Script
        SCRIPT [DEFER status time] [DEBUG filename type] PRE <NodeName | ALL_NODES> ExecutableName [arguments]
        # POST-Script
        SCRIPT [DEFER status time] [DEBUG filename type] POST <NodeName | ALL_NODES> ExecutableName [arguments]
        # HOLD-Script
        SCRIPT [DEFER status time] [DEBUG filename type] HOLD <NodeName | ALL_NODES> ExecutableName [arguments]

:dag-cmd-def:`VARS` (see :ref:`Full Description<DAGMan VARS>`)
    Specify a list of **key="Value"** pairs of information to be applied to the
    specified node's jobs as referable submit macros.

    .. code-block:: condor-dagman

        VARS <NodeName | ALL_NODES> [PREPEND | APPEND] macroname="string" [macroname2="string2" ... ]

Special Nodes
^^^^^^^^^^^^^

:dag-cmd-def:`FINAL` (see :ref:`Full Description<final-node>`)
    Create a DAG node guaranteed to run at the end of a DAG regardless
    of successful or failed execution.

    .. code-block:: condor-dagman

        FINAL NodeName SubmitDescription [DIR directory] [NOOP]

:dag-cmd-def:`PROVISIONER` (see :ref:`Full Description<DAG Provisioner Node>`)
    Create a DAG node responsible for provisioning resources to be utilized by other
    DAG nodes. Guaranteed to start before all other nodes.

    .. code-block:: condor-dagman

        PROVISIONER NodeName SubmitDescription

:dag-cmd-def:`SERVICE` (see :ref:`Full Description<DAG Service Node>`)
    Create a DAG node for specialized management/monitoring tasks. All service nodes
    are submitted prior to normal nodes.

    .. code-block:: condor-dagman

        SERVICE NodeName SubmitDescription

Throttling
^^^^^^^^^^

:dag-cmd-def:`CATEGORY` (see :ref:`Full Description<DAG throttling cmds>`)
    Assign a specified node to a DAG category.

    .. code-block:: condor-dagman

        CATEGORY <NodeName | ALL_NODES> CategoryName

:dag-cmd-def:`MAXJOBS` (see :ref:`Full Description<DAG throttling cmds>`)
    Set the max number of submitted list of jobs for a specified :dag-cmd:`CATEGORY`

    .. code-block:: condor-dagman

        MAXJOBS CategoryName MaxJobsValue

DAG Control
^^^^^^^^^^^

:dag-cmd-def:`ABORT-DAG-ON` (see :ref:`Full Description<abort-dag-on>`)
    Inform DAGMan to write a rescue file and exit when specified node exits with
    the specified value.

    .. code-block:: condor-dagman

        ABORT-DAG-ON <NodeName | ALL_NODES> AbortExitValue [RETURN DAGReturnValue]

:dag-cmd-def:`CONFIG` (see :ref:`Full Description<Per DAG Config>`)
    Specify custom DAGMan configuration file for DAGMan.

    .. code-block:: condor-dagman

        CONFIG filename

:dag-cmd-def:`ENV` (see :ref:`Full Description<DAG ENV cmd>`)
    Modify the DAGMan proper job's environment by explicitly setting environment
    variables or filtering variables from the :tool:`condor_submit_dag`\ s environment
    at submit time.

    .. code-block:: condor-dagman

        ENV GET VAR-1 [VAR-2 ... ]
        #  or
        ENV SET Key=Value;Key=Value; ...

:dag-cmd-def:`SET_JOB_ATTR` (see :ref:`Full Description<DAG set-job-attrs>`)
    Set a ClassAd attribute in the DAGMan proper job's ad.

    .. code-block:: condor-dagman

        SET_JOB_ATTR AttributeName = AttributeValue

:dag-cmd-def:`REJECT`
    Mark the DAG description file as rejected to prevent execution.

    .. code-block:: condor-dagman

        REJECT

Special Files
^^^^^^^^^^^^^

:dag-cmd-def:`DOT` (see :ref:`Full Description<visualizing-dags-with-dot>`)
    Inform DAGMan to produce a Graphviz Dot file for visualizing a DAG.

    .. code-block:: condor-dagman

        DOT filename [UPDATE | DONT-UPDATE] [OVERWRITE | DONT-OVERWRITE] [INCLUDE <dot-file-header>]

:dag-cmd-def:`JOBSTATE_LOG` (see :ref:`Full Description<DAGMan Machine Readable History>`)
    Inform DAGMan to produce a machine-readable event history file.

    .. code-block:: condor-dagman

        JOBSTATE_LOG filename

:dag-cmd-def:`NODE_STATUS_FILE` (see :ref:`Full Description<node-status-file>`)
    Inform DAGMan to produce a snapshot status file for the DAG nodes.

    .. code-block:: condor-dagman

        NODE_STATUS_FILE filename [minimumUpdateTime] [ALWAYS-UPDATE]

:dag-cmd-def:`SAVE_POINT_FILE` (see :ref:`Full Description<DAG Save Files>`)
    Inform DAGMan to write a save file the first time the specified node starts.

    .. code-block:: condor-dagman

        SAVE_POINT_FILE NodeName [Filename]

:index:`DAGMan Files<single: DAGMan; DAGMan Files>`

Produced Files
--------------

The following files are always produced automatically by DAGMan on execution. Where the
primary DAG is the only or first DAG file specified at submit time.

#. :tool:`condor_dagman` scheduler universe job files:
    .. parsed-literal::

        <Primary DAG>.condor.sub | DAGMan proper jobs submit description file.
        <Primary DAG>.dagman.log | DAGMan proper jobs event :subcom:`log` file.
        <Primary DAG>.lib.out    | DAGMan proper jobs :subcom:`output` file.
        <Primary DAG>.lib.err    | DAGMan proper jobs :subcom:`error` file.

#. DAGMan informational files:
    .. parsed-literal::

        <Primary DAG>.dagman.out | DAGMan processes debug log file.
        <Primary DAG>.nodes.log  | Shared job event log file for all jobs managed by DAGMan (Heart of DAGMan).
        <Primary DAG>.metrics    | JSON formatted file containing DAGMan metrics outputted at DAGMan exit.
#. Other:
    .. parsed-literal::

        <Primary DAG>.rescue<XXX> | Rescue DAG file denoting completed work from previous execution (see :ref:`Rescue DAG`).
        <Primary DAG>.lock        | DAGMan process lock file to prevent multiple executions of one DAG in the same directory.

Referable DAG Information
-------------------------

DAGMan provides various pieces of DAG information to scripts and jobs in the
form of special referable macros and job ClassAd attributes.

Job Macros
^^^^^^^^^^

Macros referable by job submit description as ``$(<macro>)``

.. parsed-literal::

    **JOB**              | Name of the node this job is associated with.
    **RETRY**            | Current node retry attempt value. Set to 0 on first execution.
    **FAILED_COUNT**     | Number of failed nodes currently in the DAG (intended for Final Node).
    **DAG_STATUS**       | Current :ad-attr:`DAG_Status` (intended for Final Node).
    **DAGManJobId**      | The job(s) :ad-attr:`DAGManJobId`.
    **DAG_PARENT_NAMES** | Comma separated list of node names that are parents of the node this job belongs.

Job ClassAd Attributes
^^^^^^^^^^^^^^^^^^^^^^

ClassAd attributes added to the job ad of all jobs managed by DAGMan.

.. parsed-literal::

    :ad-attr:`DAGManJobId`        | Job-Id of the DAGMan job that submitted this job.
    :ad-attr:`DAGNodeName`        | The node name of which this job belongs.
    :ad-attr:`DAGManNodeRetry`    | The nodes current retry number. First execution is 0.\
     This is only included if :macro:`DAGMAN_NODE_RECORD_INFO` includes ``Retry``.
    :ad-attr:`DAGParentNodeNames` | List of parent node names. Note depending on the number\
     of parent nodes this may be left empty.
    :ad-attr:`DAG_Status`         | Current DAG status (Intended for Final Nodes).

Script Macros
^^^^^^^^^^^^^

Macros that can be passed to a script as optional arguments like ``$<macro>``

.. parsed-literal::

    For All Scripts:
        **NODE**              | Name of the node this script is associated with.
        **RETRY**             | The node's current retry number. Set to 0 on first execution.
        **MAX_RETRIES**       | Maximum number of retries allowed for the node.
        **NODE_COUNT**        | The total number of nodes in the DAG (Including the :dag-cmd:`FINAL` node).
        **QUEUED_COUNT**      | The current number of nodes running jobs in the DAG.
        **DONE_COUNT**        | The current number of successfully completed nodes in the DAG.
        **FAILED_COUNT**      | The current number of failed nodes in the DAG.
        **FUTILE_COUNT**      | The current number of nodes that will never start in the DAG.
        **DAGID**             | The node's associated :ad-attr:`DAGManJobId`
        **DAG_STATUS**        | The current :ad-attr:`DAG_Status`.
    Only for POST Scripts:
        **CLUSTERID**         | The :ad-attr:`ClusterId` of the list of jobs associated with the node.
        **JOBID**             | The Job-ID (:ad-attr:`ClusterId` & :ad-attr:`ProcId`) of the last job in\
     the node's associated list of jobs.
        **JOB_COUNT**         | The total number of jobs associated with the node.
        **JOB_ABORT_COUNT**   | The number of jobs associated with the node that got an abort event.
        **SUCCESS**           | A boolean string that represents whether the node has been successful\
        up to this point (PRE script and list of jobs succeeded) (``True`` or ``False``).
        **RETURN**            | The exit code of the first failed job in the set or 0 for a\
     successful list of jobs execution.
        **EXIT_CODES**        | An ordered comma separated list of all :ad-attr:`ExitCode`\ s returned by\
     jobs associated with the node.
        **EXIT_CODE_LIST**    | A comma separated list of each jobs :ad-attr:`ExitCode` ordered by :ad-attr:`ProcId`\
     Removed jobs will have no value.
        **EXIT_CODE_COUNTS**  | An ordered comma separated list of the number of jobs that exited with a particular\
     :ad-attr:`ExitCode` (``{ExitCode}:{Count}``).
        **PRE_SCRIPT_RETURN** | Return value of the associated node's PRE Script.


DAG Submission and Management
-----------------------------

.. sidebar:: Tip for Querying All Jobs in a DAG

    When doing job queries to the AP queue or history, the constraint
    **-const "DAGManJobId==<DAG Job Id>"** can be used to return job
    ads for only the jobs submitted and managed by the specified DAG.

    **<DAG Job Id>** should be replaced with the :ad-attr:`ClusterId`
    of the DAGMan proper job.

For more in depth explanation of controlling a DAG see :ref:`DAG controls`

DAG Submission
^^^^^^^^^^^^^^

To submit a DAGMan workflow simply use :tool:`condor_submit_dag` on a DAG
description file.

.. code-block:: console

    $ condor_submit_dag diamond.dag

In the case of re-executing a DAG in its entirety that had previously been
run, forcibly re-submit the DAG.

.. code-block:: console

    $ condor_submit_dag -f diamond.dag

.. note::

    Forcibly re-executing a DAG will cause previous DAG informational files
    to be removed and rename any Rescue Files to ``*.old`` to invalidate them.

DAG Monitoring
^^^^^^^^^^^^^^

All the jobs managed by DAGMan and the DAGMan proper job itself can be monitored
with the tools listed below. :tool:`condor_q` by default returns a condensed overview
of jobs managed by DAGMan currently in the queue. To see all jobs individually use
the **-nobatch** flag.

+-----------------------------+-----------------------------+-----------------------------+
| :tool:`condor_q`            | :tool:`condor_watch_q`      | :tool:`htcondor dag status` |
+-----------------------------+-----------------------------+-----------------------------+

Stopping a DAG
^^^^^^^^^^^^^^

Pause/Restart
    A DAG can temporarily be stopped by using :tool:`condor_hold` on the DAGMan
    proper job. To restart the DAG simply use :tool:`condor_release`.
Halt/Resume
    A DAG can be informed to not start new work by using :tool:`htcondor dag halt`
    on the DAGMan proper job. To resume new work in the DAG simply use
    :tool:`htcondor dag resume`.
Remove
    To remove a DAG simply use :tool:`condor_rm` on the DAGMan proper job.

DAG Completion
--------------

DAGMan exits the Access Point upon successfully completing or when it can no longer
make forward progress. The latter case occurs when one or more nodes in the DAG fail
to complete successfully (see :ref:`DAGMan Node Success/Failure<DAG node success>`).

Resubmitting A Failed DAG
^^^^^^^^^^^^^^^^^^^^^^^^^

Upon failure, DAGMan will write a 'Rescue File' in the form of ``<DAG file>.rescueXXX``
e.g. ``diamond.dag.rescue001`` (see :ref:`Rescue DAG`). When resubmitted, DAGMan will
restore state from the Rescue File and skip already successfully completed nodes.

Simply run the same submit command to rescue the DAG:

.. code-block:: console

    $ condor_submit_dag diamond.dag

.. note::

    DAGMan will automatically find the most recent Rescue DAG file to restore state.

DAG Save Files
^^^^^^^^^^^^^^

Similar to Rescue DAGs, DAGMan can be re-executed from a :ref:`DAG save file<DAG Save Files>`
to restore state and skip over already successfully completed nodes.

Simply re-submit the DAG specifying a save file:

.. code-block:: console

    $ condor_submit_dag -load_save dag-progress.save diamond.dag
