[bogus section]

This section is not used by PRTE code.  But we have to put a RST
section title in this file somewhere, or Sphinx gets unhappy.  So we
put it in a section that is ignored by PRTE code.


Hello, world
============

[version]

%s (%s) %s

%s

[usage]

%s (%s) %s

Usage: %s [OPTION]...

Initiate an instance of the PMIx Reference RTE (PRRTE) DVM

The following list of command line options are available. Note that
more detailed help for any option can be obtained by adding that
option to the help request as "--help <option>".

+----------------------+-----------------------------------------------+
|                      | General Options                               |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "-h" | "--help"      | This help message                             |
+----------------------+-----------------------------------------------+
| "-h" | "--help       | Help for the specified option                 |
| <arg0>"              |                                               |
+----------------------+-----------------------------------------------+
| "-v" | "--verbose"   | Enable typical debug options                  |
+----------------------+-----------------------------------------------+
| "-V" | "--version"   | Print version and exit                        |
+----------------------+-----------------------------------------------+

+----------------------+-----------------------------------------------+
|                      | Debug Options                                 |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "--debug-daemons"    | Debug daemons - if not set, any "verbose"     |
|                      | settings will be limited to the DVM           |
|                      | controller to reduce clutter                  |
+----------------------+-----------------------------------------------+
| "--debug-daemons-    | Enable debugging of any PRRTE daemons used by |
| file"                | this application, storing their verbose       |
|                      | output in files                               |
+----------------------+-----------------------------------------------+
| "--leave-session-    | Do not discard stdout/stderr of remote PRRTE  |
| attached"            | daemons                                       |
+----------------------+-----------------------------------------------+
| "--display <arg0>"   | Comma-delimited list of options for           |
|                      | displaying information                        |
+----------------------+-----------------------------------------------+
| "--no-aggregate-help" | Do not aggregate help output from multiple   |
|                      | processes.                                    |
+----------------------+-----------------------------------------------+
| "--test-suicide      | Suicide instead of clean abort after          |
| <arg0>"              | specified delay                               |
+----------------------+-----------------------------------------------+

+----------------------+-----------------------------------------------+
|                      | DVM Options                                   |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "--runtime-options   | Comma-delimited list of runtime directives    |
| <options>"           | for the job (e.g., show progress reports on   |
|                      | DVM startup for large systems)                |
+----------------------+-----------------------------------------------+
| "--default-hostfile  | Provide a default hostfile                    |
| <filename>"          |                                               |
+----------------------+-----------------------------------------------+
| "-H|--host           | List of hosts to use for the DVM              |
| <hostname>"          |                                               |
+----------------------+-----------------------------------------------+
| "--hostfile          | Provide a hostfile                            |
| <filename>"          |                                               |
+----------------------+-----------------------------------------------+
| "--machinefile       | Provide a hostfile (synonym for "--hostfile") |
| <filename>"          |                                               |
+----------------------+-----------------------------------------------+
| "--pmixmca <key>     | Pass PMIx MCA parameters                      |
| <value>"             |                                               |
+----------------------+-----------------------------------------------+
| "--prtemca <key>     | Pass PRTE MCA parameters to the DVM           |
| <value>"             |                                               |
+----------------------+-----------------------------------------------+
| "--show-progress"    | Output a brief periodic report on DVM startup |
|                      | progress                                      |
+----------------------+-----------------------------------------------+
| "-x                  | Export an environment variable, optionally    |
| <var_name>[=value]"  | specifying a value                            |
+----------------------+-----------------------------------------------+
| "--allow-run-as-     | Allow execution as root (**STRONGLY           |
| root"                | DISCOURAGED**)                                |
+----------------------+-----------------------------------------------+
| "--daemonize"        | Daemonize the DVM daemons into the background |
+----------------------+-----------------------------------------------+
| "--forward-signals   | Comma-delimited list of additional signals    |
| <signals>"           | (names or integers) to forward                |
+----------------------+-----------------------------------------------+
| "--keepalive <arg0>" | Pipe to monitor - DVM will terminate upon     |
|                      | closure                                       |
+----------------------+-----------------------------------------------+
| "--launch-agent      | Name of daemon executable used to start       |
| <executable>"        | processes on remote nodes (default: "prted")  |
+----------------------+-----------------------------------------------+
| "--max-vm-size       | Max number of daemons to start                |
| <size>"              |                                               |
+----------------------+-----------------------------------------------+
| "--no-ready-msg"     | Do not print a "DVM ready" message            |
+----------------------+-----------------------------------------------+
| "--noprefix"         | Disable automatic "--prefix" behavior         |
+----------------------+-----------------------------------------------+
| "--prefix <dir>"     | Prefix to be used to look for RTE executables |
|                      | AND their libraries on remote nodes. Note     |
|                      | that an assumption is made that the libraries |
|                      | will be located at the same subdirectory as   |
|                      | per the configuration options given when      |
|                      | PRRTE was built.                              |
+----------------------+-----------------------------------------------+
| "--pmix-prefix <dir>" | Prefix to be used to look for the PMIx       |
|                      | library used by RTE executables on remote     |
|                      | nodes. This is the location of the top-level  |
|                      | directory for the installation.               |
+----------------------+-----------------------------------------------+
| "--report-pid <arg>" | Print out PID on stdout ("-"), stderr ("+"),  |
|                      | or a file (anything else)                     |
+----------------------+-----------------------------------------------+
| "--report-uri <arg>" | Print out URI on stdout ("-"), stderr ("+"),  |
|                      | or a file (anything else)                     |
+----------------------+-----------------------------------------------+
| "--set-sid"          | Direct the DVM daemons to separate from the   |
|                      | current session                               |
+----------------------+-----------------------------------------------+
| "--singleton <id>"   | ID of the singleton process that started us   |
+----------------------+-----------------------------------------------+
| "--system-server"    | Start the DVM as the system server            |
+----------------------+-----------------------------------------------+
| "--tmpdir <dir>"     | Set the filesystem root for the session       |
|                      | directory tree                                |
+----------------------+-----------------------------------------------+
| "--tune <arg0>"      | File(s) containing MCA params for tuning DVM  |
|                      | operations                                    |
+----------------------+-----------------------------------------------+
| "--timeout           | Timeout DVM startup if time exceeds the       |
| <seconds>"           | specified number of seconds                   |
+----------------------+-----------------------------------------------+
| "--hetero-nodes"     | The allocation contains multiple topologies,  |
|                      | so optimize the launch for that scenario. For |
|                      | example, the scheduler could be allocating    |
|                      | individual CPUs instead of entire nodes, thus |
|                      | effectively creating different topologies     |
|                      | (due to differing allocated CPUs) on each     |
|                      | node.                                         |
+----------------------+-----------------------------------------------+

Report bugs to %s

[prtemca]

Pass a PRRTE MCA parameter.

Syntax: "--prtemca <key> <value>", where "key" is the parameter name
and "value" is the parameter value.

[pmixmca]

Pass a PMIx MCA parameter

Syntax: "--pmixmca <key> <value>", where "key" is the parameter name
and "value" is the parameter value.

[tune]

Comma-delimited list of one or more files containing PRRTE and PMIx
MCA params for tuning DVM and/or application operations. Parameters in
the file will be treated as *generic* parameters and subject to the
translation rules/uncertainties.  See "--help mca" for more
information.

Syntax in the file is:

   param = value

with one parameter and its associated value per line. Empty lines and
lines beginning with the "#" character are ignored, as is any
whitespace around the "=" character.
#
[hetero-nodes]
The allocation contains multiple topologies, so optimize the launch for
that scenario. For example, the scheduler could be allocating individual
CPUs instead of entire nodes, thus effectively creating different topologies
(due to differing allocated CPUs) on each node.
#
[no-ready-msg]

Do not print a DVM ready message

[daemonize]

Daemonize the DVM daemons and controller into the background

[system-server]

Start the DVM controller and its daemons as the system server on their
nodes

[set-sid]

Direct the DVM (controller and daemons) to separate from the current
session

[report-pid]

Printout DVM controller's PID on stdout [-], stderr [+], or a file
[anything else]

[report-uri]

Printout DVM controller's URI on stdout [-], stderr [+], or a file
[anything else]

[test-suicide]

Test DVM cleanup upon daemon failure by having one daemon suicide
after delay

[default-hostfile]

Specify a default hostfile.

Also see "--help hostfile".

[singleton]

DVM is being started by a singleton process (i.e., one not started by
a DVM) - the argument must be the PMIx ID of the singleton process
that started us

[keepalive]

Pipe for DVM controller to monitor - DVM will terminate upon closure

[launch-agent]

Name of daemon executable used to start processes on remote nodes
(default: prted).  This is the executable the DVM controller shall
start on each remote node when establishing the DVM.

[max-vm-size]

Maximum number of daemons to start - sets the maximum size of the DVM.

[debug-daemons]

Debug daemon output enabled. This is a somewhat limited stream of
information normally used to simply confirm that the daemons started.
Includes leaving the output streams open.

[debug-daemons-file]

Debug daemon output is enabled and all output from the daemons is
redirected into files with names of the form:

   output-prted-<daemon-nspace>-<nodename>.log

These names avoid conflict on shared file systems. The files are
located in the top-level session directory assigned to the DVM.

See the "Session directory" HTML documentation for additional details
about the PRRTE session directory.

[leave-session-attached]

Do not discard stdout/stderr of remote PRRTE daemons. The primary use
for this option is to ensure that the daemon output streams (i.e.,
stdout and stderr) remain open after launch, thus allowing the user to
see any daemon-generated error messages. Otherwise, the daemon will
"daemonize" itself upon launch, thereby closing its output streams.

[tmpdir]

Define the root location for the PRRTE session directory tree

See the "Session directory" HTML documentation for additional details
about the PRRTE session directory.

[prefix]

Prefix to be used to look for PRRTE executables. PRRTE automatically
sets the prefix for remote daemons if it was either configured with
the "--enable-prte-prefix-by-default" option OR prte itself was
executed with an absolute path to the prte command. This option
overrides those settings, if present, and forces use of the provided
path.

[noprefix]

Disable automatic "--prefix" behavior. PRRTE automatically sets the
prefix for remote daemons if it was either configured with the "--
enable-prte-prefix-by-default" option OR prte itself was executed with
an absolute path to the "prte" command. This option disables that
behavior.

[pmix-prefix]

Prefix to be used by a PRRTE executable to look for its PMIx installation
on remote nodes. This is the location of the top-level directory for the
installation. If the installation has not been moved, it would be the
value given to "--prefix" when the installation was configured.

Note that PRRTE cannot determine the exact name of the library subdirectory
under this location. For example, some systems will call it "lib" while others
call it "lib64". Accordingly, PRRTE will use the library subdirectory name
of the PMIx installation used to build PRRTE.

[forward-signals]

Comma-delimited list of additional signals (names or integers) to
forward to application processes ("none" = forward nothing). Signals
provided by default include SIGTSTP, SIGUSR1, SIGUSR2, SIGABRT,
SIGALRM, and SIGCONT.

[allow-run-as-root]

Allow execution as root **(STRONGLY DISCOURAGED)**.

Running as root exposes the user to potentially catastrophic file
system corruption and damage — e.g., if the user accidentally points
the root of the session directory to a system required point, this
directory and all underlying elements will be deleted upon job
completion, thereby rendering the system inoperable.

It is recognized that some environments (e.g., containers) may require
operation as root, and that the user accepts the risks in those
scenarios. Accordingly, one can override PRRTE's run-as-root
protection by providing one of the following:

* The "--allow-run-as-root" command line directive

* Adding **BOTH** of the following environmental parameters:

     * "PRTE_ALLOW_RUN_AS_ROOT=1"

     * "PRTE_ALLOW_RUN_AS_ROOT_CONFIRM=1"

Again, we recommend this only be done if absolutely necessary.

[no-aggregate-help]

Do not aggregate help output from multiple processes. PRRTE defaults
to aggregating messages generated by its "help" subsystem so that only
one is printed out per topic (along with the number of processes that
reported the same issue). This is done to avoid users receiving a
flood of one-per-process error messages, all containing the identical
error report.  Setting this option turns off the aggregation, thereby
allowing the user to see duplicate reports from multiple processes.

[timeout]

Timeout DVM startup if time exceeds the specified number of seconds.
The DVM startup will abort after the specified interval.

[x]

Export an environment variable, optionally specifying a value. For
example:

* "-x foo" exports the environment variable "foo" and takes its value
  from the current environment.

* "-x foo=bar" exports the environment variable name "foo" and sets
  its value to "bar" in the started processes.

* "-x foo*" exports all current environmental variables starting with
  "foo".

[show-progress]

Output a brief periodic report on DVM startup progress

[hostfile]

PRRTE supports several levels of user-specified host lists based on an
established precedence order. Users can specify a default hostfile
that contains a list of nodes to be used by the DVM. Only one default
hostfile can be provided for a given DVM. In addition, users can
specify a hostfile that contains a list of nodes to be used for a DVM,
or can provide a comma-delimited list of nodes to be used for that DVM
via the "--host" command line option.

The precedence order applied to these various options depends to some
extent on the local environment. The following table illustrates how
host and hostfile directives work together to define the set of hosts
upon which a DVM will execute in the absence of a resource manager
(RM):

+------------+---------+------------+----------------------------------+
| Default    | host    | hostfile   | Result                           |
| hostfile   |         |            |                                  |
|============|=========|============|==================================|
| unset      | unset   | unset      | The DVN will consist solely of   |
|            |         |            | the local host where the DVM was |
|            |         |            | started.                         |
+------------+---------+------------+----------------------------------+
| unset      | set     | unset      | Host option defines resource     |
|            |         |            | list for the DVM.                |
+------------+---------+------------+----------------------------------+
| unset      | unset   | set        | Hostfile option defines resource |
|            |         |            | list for the DVM.                |
+------------+---------+------------+----------------------------------+
| unset      | set     | set        | Hostfile option defines resource |
|            |         |            | list for the DVM, then host      |
|            |         |            | filters the list to define the   |
|            |         |            | final set of nodes to be used by |
|            |         |            | the DVM                          |
+------------+---------+------------+----------------------------------+
| set        | unset   | unset      | Default hostfile defines         |
|            |         |            | resource list for the DVM        |
+------------+---------+------------+----------------------------------+
| set        | set     | unset      | Default hostfile defines         |
|            |         |            | resource list for the DVM, then  |
|            |         |            | host filters the list to define  |
|            |         |            | the final set of nodes to be     |
|            |         |            | used by the DVM                  |
+------------+---------+------------+----------------------------------+
| set        | set     | set        | Default hostfile defines         |
|            |         |            | resource list for the DVM, then  |
|            |         |            | hostfile filters the list, and   |
|            |         |            | then host filters the list to    |
|            |         |            | define the final set of nodes to |
|            |         |            | be used by the DVM               |
+------------+---------+------------+----------------------------------+

This changes somewhat in the presence of an RM as that entity
specifies the initial allocation of nodes. In this case, the default
hostfile, hostfile and host directives are all used to filter the RM's
specification so that a user can utilize different portions of the
allocation for different DVMs. This is done according to the same
precedence order as in the prior table, with the RM providing the
initial pool of nodes.

Hostfiles (sometimes called "machine files") are a combination of two
things:

1. A listing of hosts on which to launch processes.

2. Optionally, limit the number of processes which can be launched on
   each host.

Hostfile syntax consists of one node name on each line, optionally
including a designated number of "slots":

   # This is a comment line, and will be ignored
   node01  slots=10
   node13  slots=5

   node15
   node16
   node17  slots=3
   ...

Blank lines and lines beginning with a "#" are ignored.

A "slot" is the PRRTE term for an allocatable unit where we can launch
a process.  See the section on definition of the term "slot" for a
longer description of slots.

In the absence of the "slot" parameter, PRRTE will assign either the
number of slots to be the number of CPUs detected on the node or the
resource manager-assigned value if operating in the presence of an RM.

Important:

  If using a resource manager, the user-specified number of slots is
  capped by the RM-assigned value.

See the "Host specification" HTML documentation for details about the
format and content of hostfiles.

[machinefile]

Provide a hostfile.  This option is a synonym for "--hostfile"; see
that option for more information.

[host]

Host syntax consists of a comma-delimited list of node names, each
entry optionally containing a ":N" extension indicating the number of
slots to assign to that entry:

   --host node01:5,node02

In the absence of the slot extension, one slot will be assigned to the
node. Duplicate entries are aggregated and the number of slots
assigned to that node are summed together.

Note:

  A "slot" is the PRRTE term for an allocatable unit where we can
  launch a process. Thus, the number of slots equates to the maximum
  number of processes PRRTE may start on that node without
  oversubscribing it.

[display]

The "display" command line directive must be accompanied by a comma-
delimited list of case-insensitive options indicating what information
about the job and/or allocation is to be displayed. The full directive
need not be provided — only enough characters are required to uniquely
identify the directive. For example, "ALL" is sufficient to represent
the "ALLOCATION" directive — while "MAP" can not be used to represent
"MAP-DEVEL" (though "MAP-D" would suffice).

Supported values include:

* "ALLOCATION" displays the detected hosts and slot assignments for
  this job

* "BINDINGS" displays the resulting bindings applied to processes in
  this job

* "MAP" displays the resulting locations assigned to processes in this
  job

* "MAP-DEVEL" displays a more detailed report on the locations
  assigned to processes in this job that includes local and node
  ranks, assigned bindings, and other data

* "TOPO=LIST" displays the topology of each node in the semicolon-
  delimited list that is allocated to the job

* "CPUS[=LIST]" displays the available CPUs on the provided semicolon-
  delimited list of nodes (defaults to all nodes)

The display command line directive can include qualifiers by adding a
colon (":") and any combination of one or more of the following
(delimited by colons):

* "PARSEABLE" directs that the output be provided in a format that is
  easily parsed by machines. Note that "PARSABLE" is also accepted as
  a typical spelling for the qualifier.

Provided qualifiers will apply to *all* of the display directives.

[runtime-options]

The "--runtime-options" command line directive must be accompanied by
a comma-delimited list of case-insensitive options that control the
runtime behavior of the job. The full directive need not be provided —
only enough characters are required to uniquely identify the
directive.

Runtime options are typically "true" or "false", though this is not a
requirement on developers. Since the value of each option may need to
be set (e.g., to override a default set by MCA parameter), the syntax
of the command line directive includes the use of an "=" character to
allow inclusion of a value for the option. For example, one can set
the "ABORT-NONZERO-STATUS" option to "true" by specifying it as
"ABORT-NONZERO-STATUS=1". Note that boolean options can be set to
"true" using a non-zero integer or a case-insensitive string of the
word "true".  For the latter representation, the user need only
provide at least the "T" character. The same policy applies to setting
a boolean option to "false".

Note that a boolean option will default to "true" if provided without
a value. Thus, "--runtime-options abort-nonzero" is sufficient to set
the "ABORT-NONZERO-STATUS" option to "true".

Supported values include:

* "ERROR-NONZERO-STATUS[=(bool)]": if set to false, this directs the
  runtime to treat a process that exits with non-zero status as a
  normal termination.  If set to true, the runtime will consider such
  an occurrence as an error termination and take appropriate action —
  i.e., the job will be terminated unless a runtime option directs
  otherwise. This option defaults to a true value if the option is
  given without a value.

* "DONOTLAUNCH": directs the runtime to map but not launch the
  specified job. This is provided to help explore possible process
  placement patterns before actually starting execution. No value need
  be passed as this is not an option that can be set by default in
  PRRTE.

* "SHOW-PROGRESS[=(bool)]": requests that the runtime provide progress
  reports on its startup procedure — i.e., the launch of its daemons
  in support of a job. This is typically used to debug DVM startup on
  large systems.  This option defaults to a true value if the option
  is given without a value.

* "NOTIFYERRORS[=(bool)]": if set to true, requests that the runtime
  provide a PMIx event whenever a job encounters an error — e.g., a
  process fails.  The event is to be delivered to each remaining
  process in the job. This option defaults to a true value if the
  option is given without a value.  See "--help notifications" for
  more detail as to the PMIx event codes available for capturing
  failure events.

* "RECOVERABLE[=(bool)]": if set to true, this indicates that the
  application wishes to consider the job as recoverable — i.e., the
  application is assuming responsibility for recovering from any
  process failure. This could include application-driven spawn of a
  substitute process or internal compensation for the missing process.
  This option defaults to a true value if the option is given without
  a value.

* "AUTORESTART[=(bool)]": if set to true, this requests that the
  runtime automatically restart failed processes up to "max restarts"
  number of times. This option defaults to a true value if the option
  is given without a value.

* "CONTINUOUS[=(bool)]": if set to true, this informs the runtime that
  the processes in this job are to run until explicitly terminated.
  Processes that fail are to be automatically restarted up to "max
  restarts" number of times. Notification of process failure is to be
  delivered to all processes in the application. This is the
  equivalent of specifying "RECOVERABLE", "NOTIFYERRORS", and
  "AUTORESTART" options except that the runtime, not the application,
  assumes responsibility for process recovery. This option defaults to
  a true value if the option is given without a value.

* "MAX-RESTARTS=<int>": indicates the maximum number of times a given
  process is to be restarted. This can be set at the application or
  job level (which will then apply to all applications in that job).

* "EXEC-AGENT=<path>" indicates the executable that shall be used to
  start an application process. The resulting command for starting an
  application process will be "<path> app <app-argv>". The path may
  contain its own command line arguments.

* "DEFAULT-EXEC-AGENT": directs the runtime to use the system default
  exec agent to start an application process. No value need be passed
  as this is not an option that can be set by default in PRRTE.

* "OUTPUT-PROCTABLE[(=channel)]": directs the runtime to report the
  convential debugger process table (includes PID and host location of
  each process in the application). Output is directed to stdout if
  the channel is "-", stderr if "+", or into the specified file
  otherwise. If no channel is specified, output will be directed to
  stdout.

* "STOP-ON-EXEC": directs the runtime to stop the application
  process(es) immediately upon exec'ing them. The directive will apply
  to all processes in the job.

* "STOP-IN-INIT": indicates that the runtime should direct the
  application process(es) to stop in "PMIx_Init()". The directive will
  apply to all processes in the job.

* "STOP-IN-APP": indicates that the runtime should direct application
  processes to stop at some application-defined place and notify they
  are ready-to-debug. The directive will apply to all processes in the
  job.

* "TIMEOUT=<string>": directs the runtime to terminate the job after
  it has executed for the specified time. Time is specified in colon-
  delimited format — e.g., "01:20:13:05" to indicate 1 day, 20 hours,
  13 minutes and 5 seconds. Time specified without colons will be
  assumed to have been given in seconds.

* "SPAWN-TIMEOUT=<string>": directs the runtime to terminate the job
  if job launch is not completed within the specified time. Time is
  specified in colon-delimited format — e.g., "01:20:13:05" to
  indicate 1 day, 20 hours, 13 minutes and 5 seconds.  Time specified
  without colons will be assumed to have been given in seconds.

* "REPORT-STATE-ON-TIMEOUT[(=bool)]": directs the runtime to provide a
  detailed report on job and application process state upon job
  timeout. This option defaults to a true value if the option is given
  without a value.

* "GET-STACK-TRACES[(=bool)]": requests that the runtime provide stack
  traces on all application processes still executing upon timeout.
  This option defaults to a true value if the option is given without
  a value.

* "REPORT-CHILD-JOBS-SEPARATELY[(=bool)]": directs the runtime to
  report the exit status of any child jobs spawned by the primary job
  separately. If false, then the final exit status reported will be
  zero if the primary job and all spawned jobs exit normally, or the
  first non-zero status returned by either primary or child jobs. This
  option defaults to a true value if the option is given without a
  value.

* "AGGREGATE-HELP-MESSAGES[(=bool)]": directs the runtime to aggregate
  help messages, reporting each unique help message once accompanied
  by the number of processes that reported it. This option defaults to
  a true value if the option is given without a value.

* "FWD-ENVIRONMENT[(=bool)]": directs the runtime to forward the
  entire local environment in support of the application. This option
  defaults to a true value if the option is given without a value.

The "--runtime-options" command line option has no qualifiers.

Note:

  Directives are case-insensitive.  "FWD-ENVIRONMENT" is the same as
  "fwd-environment".
