Title: | Crew Launcher Plugins for Traditional High-Performance Computing Clusters |
Description: | In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The 'crew.cluster' package extends the 'mirai'-powered 'crew' package with worker launcher plugins for traditional high-performance computing systems. Inspiration also comes from packages 'mirai' by Gao (2023) https://github.com/r-lib/mirai, 'future' by Bengtsson (2021) <doi:10.32614/RJ-2021-048>, 'rrq' by FitzJohn and Ashton (2023) https://github.com/mrc-ide/rrq, 'clustermq' by Schubert (2019) <doi:10.1093/bioinformatics/btz284>), and 'batchtools' by Lang, Bischl, and Surmann (2017). <doi:10.21105/joss.00135>. |
Version: | 0.3.8 |
License: | MIT + file LICENSE |
URL: | https://wlandau.github.io/crew.cluster/, https://github.com/wlandau/crew.cluster |
BugReports: | https://github.com/wlandau/crew.cluster/issues |
Depends: | R (≥ 4.0.0) |
Imports: | crew (≥ 1.2.0), ps, lifecycle, R6, rlang, utils, vctrs, xml2, yaml |
Suggests: | knitr (≥ 1.30), markdown (≥ 1.1), rmarkdown (≥ 2.4), testthat (≥ 3.0.0) |
Encoding: | UTF-8 |
Language: | en-US |
Config/testthat/edition: | 3 |
RoxygenNote: | 7.3.2 |
NeedsCompilation: | no |
Packaged: | 2025-06-09 14:21:13 UTC; C240390 |
Author: | William Michael Landau
|
Maintainer: | William Michael Landau <will.landau.oss@gmail.com> |
Repository: | CRAN |
Date/Publication: | 2025-06-09 14:40:02 UTC |
crew.cluster: crew launcher plugins for traditional high-performance computing clusters
Description
In computationally demanding analysis projects,
statisticians and data scientists asynchronously
deploy long-running tasks to distributed systems,
ranging from traditional clusters to cloud services.
The crew.cluster
package extends the
mirai
-powered
crew
package with worker
launcher plugins for traditional
high-performance computing systems.
Inspiration also comes from packages
mirai
,
future
,
rrq
,
clustermq
,
and batchtools
.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
Abstract cluster launcher class
Description
R6
class to help develop specific cluster launcher plugins.
Details
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
Super class
crew::crew_class_launcher
-> crew_class_launcher_cluster
Active bindings
options_cluster
Methods
Public methods
Inherited methods
crew::crew_class_launcher$call()
crew::crew_class_launcher$crashes()
crew::crew_class_launcher$launch()
crew::crew_class_launcher$poll()
crew::crew_class_launcher$resolve()
crew::crew_class_launcher$scale()
crew::crew_class_launcher$set_name()
crew::crew_class_launcher$settings()
crew::crew_class_launcher$start()
crew::crew_class_launcher$terminate()
crew::crew_class_launcher$terminate_workers()
crew::crew_class_launcher$update()
Method new()
Abstract launcher constructor.
Usage
crew_class_launcher_cluster$new( name = NULL, workers = NULL, seconds_interval = NULL, seconds_timeout = NULL, seconds_launch = NULL, seconds_idle = NULL, seconds_wall = NULL, tasks_max = NULL, tasks_timers = NULL, reset_globals = NULL, reset_packages = NULL, reset_options = NULL, garbage_collection = NULL, tls = NULL, r_arguments = NULL, options_metrics = NULL, options_cluster = NULL )
Arguments
name
workers
seconds_interval
seconds_timeout
seconds_launch
seconds_idle
seconds_wall
tasks_max
tasks_timers
reset_globals
reset_packages
reset_options
garbage_collection
tls
r_arguments
options_metrics
options_cluster
Returns
An abstract launcher object.
Method validate()
Validate the launcher.
Usage
crew_class_launcher_cluster$validate()
Returns
NULL
(invisibly). Throws an error if a field is invalid.
Method launch_worker()
Launch a local process worker which will dial into a socket.
Usage
crew_class_launcher_cluster$launch_worker( call, name, launcher, worker, instance = NULL )
Arguments
call
Character string, a namespaced call to
crew::crew_worker()
which will run in the worker and accept tasks.name
Character string, an informative worker name.
launcher
Character string, name of the launcher.
worker
Character string, name of the worker instance.
instance
Deprecated in
crew.cluster
Details
The call
argument is R code that will run to
initiate the worker.
Returns
A handle object to allow the termination of the worker later on.
Method terminate_worker()
Terminate a local process worker.
Usage
crew_class_launcher_cluster$terminate_worker(handle)
Arguments
handle
A process handle object previously returned by
launch_worker()
.
Returns
NULL
(invisibly).
See Also
Other cluster:
crew_class_monitor_cluster
,
crew_launcher_cluster()
,
crew_monitor_cluster()
,
crew_options_cluster()
LSF launcher class
Description
R6
class to launch and manage LSF workers.
Details
See crew_launcher_lsf()
.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
Super classes
crew::crew_class_launcher
-> crew.cluster::crew_class_launcher_cluster
-> crew_class_launcher_lsf
Methods
Public methods
Inherited methods
crew::crew_class_launcher$call()
crew::crew_class_launcher$crashes()
crew::crew_class_launcher$launch()
crew::crew_class_launcher$poll()
crew::crew_class_launcher$resolve()
crew::crew_class_launcher$scale()
crew::crew_class_launcher$set_name()
crew::crew_class_launcher$settings()
crew::crew_class_launcher$start()
crew::crew_class_launcher$terminate()
crew::crew_class_launcher$terminate_workers()
crew::crew_class_launcher$update()
crew.cluster::crew_class_launcher_cluster$initialize()
crew.cluster::crew_class_launcher_cluster$launch_worker()
crew.cluster::crew_class_launcher_cluster$terminate_worker()
Method validate()
Validate the launcher.
Usage
crew_class_launcher_lsf$validate()
Returns
NULL
(invisibly). Throws an error if a field is invalid.
Method script()
Generate the job script.
Usage
crew_class_launcher_lsf$script(name)
Arguments
name
Character of length 1, name of the job. For inspection purposes, you can supply a mock job name.
Details
Includes everything except the worker-instance-specific
job name and the worker-instance-specific
call to crew::crew_worker()
, both of which get inserted at
the bottom of the script at launch time.
Returns
Character vector of the lines of the job script.
Examples
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) { launcher <- crew_launcher_lsf( lsf_cwd = getwd(), lsf_log_output = "log_file_%J.log", lsf_log_error = NULL, lsf_memory_gigabytes_limit = 4 ) launcher$script(name = "my_job_name") }
See Also
Other lsf:
crew_controller_lsf()
,
crew_launcher_lsf()
,
crew_options_lsf()
Examples
## ------------------------------------------------
## Method `crew_class_launcher_lsf$script`
## ------------------------------------------------
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
launcher <- crew_launcher_lsf(
lsf_cwd = getwd(),
lsf_log_output = "log_file_%J.log",
lsf_log_error = NULL,
lsf_memory_gigabytes_limit = 4
)
launcher$script(name = "my_job_name")
}
PBS/TORQUE launcher class
Description
R6
class to launch and manage PBS/TORQUE workers.
Details
See crew_launcher_pbs()
.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
Super classes
crew::crew_class_launcher
-> crew.cluster::crew_class_launcher_cluster
-> crew_class_launcher_pbs
Methods
Public methods
Inherited methods
crew::crew_class_launcher$call()
crew::crew_class_launcher$crashes()
crew::crew_class_launcher$launch()
crew::crew_class_launcher$poll()
crew::crew_class_launcher$resolve()
crew::crew_class_launcher$scale()
crew::crew_class_launcher$set_name()
crew::crew_class_launcher$settings()
crew::crew_class_launcher$start()
crew::crew_class_launcher$terminate()
crew::crew_class_launcher$terminate_workers()
crew::crew_class_launcher$update()
crew.cluster::crew_class_launcher_cluster$initialize()
crew.cluster::crew_class_launcher_cluster$launch_worker()
crew.cluster::crew_class_launcher_cluster$terminate_worker()
Method validate()
Validate the launcher.
Usage
crew_class_launcher_pbs$validate()
Returns
NULL
(invisibly). Throws an error if a field is invalid.
Method script()
Generate the job script.
Usage
crew_class_launcher_pbs$script(name)
Arguments
name
Character of length 1, name of the job. For inspection purposes, you can supply a mock job name.
Details
Includes everything except the worker-instance-specific
job name and the worker-instance-specific
call to crew::crew_worker()
, both of which get inserted at
the bottom of the script at launch time.
Returns
Character vector of the lines of the job script.
Examples
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) { launcher <- crew_launcher_pbs( pbs_cores = 2, pbs_memory_gigabytes_required = 4 ) launcher$script(name = "my_job_name") }
See Also
Other pbs:
crew_controller_pbs()
,
crew_launcher_pbs()
,
crew_options_pbs()
Examples
## ------------------------------------------------
## Method `crew_class_launcher_pbs$script`
## ------------------------------------------------
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
launcher <- crew_launcher_pbs(
pbs_cores = 2,
pbs_memory_gigabytes_required = 4
)
launcher$script(name = "my_job_name")
}
SGE launcher class
Description
R6
class to launch and manage SGE workers.
Details
See crew_launcher_sge()
.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
Super classes
crew::crew_class_launcher
-> crew.cluster::crew_class_launcher_cluster
-> crew_class_launcher_sge
Methods
Public methods
Inherited methods
crew::crew_class_launcher$call()
crew::crew_class_launcher$crashes()
crew::crew_class_launcher$launch()
crew::crew_class_launcher$poll()
crew::crew_class_launcher$resolve()
crew::crew_class_launcher$scale()
crew::crew_class_launcher$set_name()
crew::crew_class_launcher$settings()
crew::crew_class_launcher$start()
crew::crew_class_launcher$terminate()
crew::crew_class_launcher$terminate_workers()
crew::crew_class_launcher$update()
crew.cluster::crew_class_launcher_cluster$initialize()
crew.cluster::crew_class_launcher_cluster$launch_worker()
crew.cluster::crew_class_launcher_cluster$terminate_worker()
Method validate()
Validate the launcher.
Usage
crew_class_launcher_sge$validate()
Returns
NULL
(invisibly). Throws an error if a field is invalid.
Method script()
Generate the job script.
Usage
crew_class_launcher_sge$script(name)
Arguments
name
Character of length 1, name of the job. For inspection purposes, you can supply a mock job name.
Details
Includes everything except the worker-instance-specific
job name and the worker-instance-specific
call to crew::crew_worker()
, both of which get inserted at
the bottom of the script at launch time.
Returns
Character vector of the lines of the job script.
Examples
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) { launcher <- crew_launcher_sge( sge_cores = 2, sge_memory_gigabytes_required = 4 ) launcher$script(name = "my_job_name") }
See Also
Other sge:
crew_class_monitor_sge
,
crew_controller_sge()
,
crew_launcher_sge()
,
crew_monitor_sge()
,
crew_options_sge()
Examples
## ------------------------------------------------
## Method `crew_class_launcher_sge$script`
## ------------------------------------------------
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
launcher <- crew_launcher_sge(
sge_cores = 2,
sge_memory_gigabytes_required = 4
)
launcher$script(name = "my_job_name")
}
SLURM launcher class
Description
R6
class to launch and manage SLURM workers.
Details
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
Super classes
crew::crew_class_launcher
-> crew.cluster::crew_class_launcher_cluster
-> crew_class_launcher_slurm
Methods
Public methods
Inherited methods
crew::crew_class_launcher$call()
crew::crew_class_launcher$crashes()
crew::crew_class_launcher$launch()
crew::crew_class_launcher$poll()
crew::crew_class_launcher$resolve()
crew::crew_class_launcher$scale()
crew::crew_class_launcher$set_name()
crew::crew_class_launcher$settings()
crew::crew_class_launcher$start()
crew::crew_class_launcher$terminate()
crew::crew_class_launcher$terminate_workers()
crew::crew_class_launcher$update()
crew.cluster::crew_class_launcher_cluster$initialize()
crew.cluster::crew_class_launcher_cluster$launch_worker()
crew.cluster::crew_class_launcher_cluster$terminate_worker()
Method validate()
Validate the launcher.
Usage
crew_class_launcher_slurm$validate()
Returns
NULL
(invisibly). Throws an error if a field is invalid.
Method script()
Generate the job script.
Usage
crew_class_launcher_slurm$script(name)
Arguments
name
Character of length 1, name of the job. For inspection purposes, you can supply a mock job name.
Details
Includes everything except the worker-instance-specific
job name and the worker-instance-specific
call to crew::crew_worker()
, both of which get inserted at
the bottom of the script at launch time.
Returns
Character vector of the lines of the job script.
Examples
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) { launcher <- crew_launcher_slurm( slurm_log_output = "log_file_%A.log", slurm_log_error = NULL, slurm_memory_gigabytes_per_cpu = 4096 ) launcher$script(name = "my_job_name") }
See Also
Other slurm:
crew_class_monitor_slurm
,
crew_controller_slurm()
,
crew_launcher_slurm()
,
crew_monitor_slurm()
,
crew_options_slurm()
Examples
## ------------------------------------------------
## Method `crew_class_launcher_slurm$script`
## ------------------------------------------------
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
launcher <- crew_launcher_slurm(
slurm_log_output = "log_file_%A.log",
slurm_log_error = NULL,
slurm_memory_gigabytes_per_cpu = 4096
)
launcher$script(name = "my_job_name")
}
Abstract cluster monitor class
Description
Abstract cluster monitor R6
class
Details
Active bindings
verbose
command_list
command_terminate
Methods
Public methods
Method new()
Abstract cluster monitor constructor.
Usage
crew_class_monitor_cluster$new( verbose = NULL, command_list = NULL, command_terminate = NULL )
Arguments
verbose
command_list
command_terminate
Returns
an abstract cluster monitor object.
Method validate()
Validate the monitor.
Usage
crew_class_monitor_cluster$validate()
Returns
NULL
(invisibly).
See Also
Other cluster:
crew_class_launcher_cluster
,
crew_launcher_cluster()
,
crew_monitor_cluster()
,
crew_options_cluster()
SGE monitor class
Description
SGE monitor R6
class
Details
See crew_monitor_sge()
.
Super class
crew.cluster::crew_class_monitor_cluster
-> crew_class_monitor_sge
Methods
Public methods
Inherited methods
Method jobs()
List SGE jobs.
Usage
crew_class_monitor_sge$jobs(user = ps::ps_username())
Arguments
user
Character of length 1, user name of the jobs to list.
Returns
A tibble
with one row per SGE job and columns with
specific details.
Method terminate()
Terminate one or more SGE jobs.
Usage
crew_class_monitor_sge$terminate(jobs = NULL, all = FALSE)
Arguments
jobs
Character vector of job names or job IDs to terminate. Ignored if
all
is set toTRUE
.all
Logical of length 1, whether to terminate all the jobs under your user name. This terminates ALL your SGE jobs, regardless of whether
crew.cluster
launched them, so use with caution!
Returns
NULL
(invisibly).
See Also
Other sge:
crew_class_launcher_sge
,
crew_controller_sge()
,
crew_launcher_sge()
,
crew_monitor_sge()
,
crew_options_sge()
SLURM monitor class
Description
SLURM monitor R6
class
Details
See crew_monitor_slurm()
.
Super class
crew.cluster::crew_class_monitor_cluster
-> crew_class_monitor_slurm
Methods
Public methods
Inherited methods
Method jobs()
List SLURM jobs.
Usage
crew_class_monitor_slurm$jobs(user = ps::ps_username())
Arguments
user
Character of length 1, user name of the jobs to list.
Details
This function loads the entire SLURM queue for all users, so it may take several seconds to execute. It is intended for interactive use, and should especially be avoided in scripts where it is called frequently. It requires SLURM version 20.02 or higher, along with the YAML plugin.
Returns
A tibble
with one row per SLURM job and columns with
specific details.
Method terminate()
Terminate one or more SLURM jobs.
Usage
crew_class_monitor_slurm$terminate(jobs = NULL, all = FALSE)
Arguments
jobs
Character vector of job names or job IDs to terminate. Ignored if
all
is set toTRUE
.all
Logical of length 1, whether to terminate all the jobs under your user name. This terminates ALL your SLURM jobs, regardless of whether
crew.cluster
launched them, so use with caution!
Returns
NULL
(invisibly).
See Also
Other slurm:
crew_class_launcher_slurm
,
crew_controller_slurm()
,
crew_launcher_slurm()
,
crew_monitor_slurm()
,
crew_options_slurm()
Create a controller with a
LSF launcher.
Description
Create an R6
object to submit tasks and
launch workers on LSF workers.
Usage
crew_controller_lsf(
name = NULL,
workers = 1L,
host = NULL,
port = NULL,
tls = crew::crew_tls(mode = "automatic"),
tls_enable = NULL,
tls_config = NULL,
serialization = NULL,
seconds_interval = 0.25,
seconds_timeout = 60,
seconds_launch = 86400,
seconds_idle = 300,
seconds_wall = Inf,
seconds_exit = NULL,
retry_tasks = NULL,
tasks_max = Inf,
tasks_timers = 0L,
reset_globals = TRUE,
reset_packages = FALSE,
reset_options = FALSE,
garbage_collection = FALSE,
crashes_error = NULL,
r_arguments = c("--no-save", "--no-restore"),
crashes_max = 5L,
backup = NULL,
options_metrics = crew::crew_options_metrics(),
options_cluster = crew.cluster::crew_options_lsf(),
verbose = NULL,
command_submit = NULL,
command_terminate = NULL,
command_delete = NULL,
script_directory = NULL,
script_lines = NULL,
lsf_cwd = NULL,
lsf_log_output = NULL,
lsf_log_error = NULL,
lsf_memory_gigabytes_limit = NULL,
lsf_memory_gigabytes_required = NULL,
lsf_cores = NULL
)
Arguments
name |
Character string, name of the launcher. If the name is
|
workers |
Maximum number of workers to run concurrently
when auto-scaling, excluding task retries and manual calls to |
host |
IP address of the |
port |
TCP port to listen for the workers. If |
tls |
A TLS configuration object from |
tls_enable |
Deprecated on 2023-09-15 in version 0.4.1.
Use argument |
tls_config |
Deprecated on 2023-09-15 in version 0.4.1.
Use argument |
serialization |
Either |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete. In certain cases, exponential
backoff is used with this argument passed to |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
seconds_exit |
Deprecated on 2023-09-21 in version 0.1.2.9000. No longer necessary. |
retry_tasks |
Deprecated on 2025-01-13 ( |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
|
reset_packages |
|
reset_options |
|
garbage_collection |
|
crashes_error |
Deprecated on 2025-01-13 ( |
r_arguments |
Optional character vector of command line arguments
to pass to |
crashes_max |
In rare cases, a worker may exit unexpectedly
before it completes its current task. If this happens,
|
backup |
An optional Limitations of |
options_metrics |
Either |
options_cluster |
An options list from
|
verbose |
Deprecated. Use |
command_submit |
Deprecated. Use |
command_terminate |
Deprecated. Use |
command_delete |
Deprecated on 2024-01-08 (version 0.1.4.9001).
Use |
script_directory |
Deprecated. Use |
script_lines |
Deprecated. Use |
lsf_cwd |
Deprecated. Use |
lsf_log_output |
Deprecated. Use |
lsf_log_error |
Deprecated. Use |
lsf_memory_gigabytes_limit |
Deprecated.
Use |
lsf_memory_gigabytes_required |
Deprecated.
Use |
lsf_cores |
Deprecated. Use |
Details
WARNING: the crew.cluster
LSF plugin is experimental
and has not actually been tested on a LSF cluster. Please proceed
with caution and report bugs to
https://github.com/wlandau/crew.cluster.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other lsf:
crew_class_launcher_lsf
,
crew_launcher_lsf()
,
crew_options_lsf()
Examples
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
controller <- crew_controller_lsf()
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()$result
controller$terminate()
}
Create a controller with a
PBS/TORQUE launcher.
Description
Create an R6
object to submit tasks and
launch workers on a PBS or TORQUE cluster.
Usage
crew_controller_pbs(
name = NULL,
workers = 1L,
host = NULL,
port = NULL,
tls = crew::crew_tls(mode = "automatic"),
tls_enable = NULL,
tls_config = NULL,
serialization = NULL,
seconds_interval = 0.25,
seconds_timeout = 60,
seconds_launch = 86400,
seconds_idle = 300,
seconds_wall = Inf,
seconds_exit = NULL,
retry_tasks = NULL,
tasks_max = Inf,
tasks_timers = 0L,
reset_globals = TRUE,
reset_packages = FALSE,
reset_options = FALSE,
garbage_collection = FALSE,
crashes_error = NULL,
r_arguments = c("--no-save", "--no-restore"),
crashes_max = 5L,
backup = NULL,
options_metrics = crew::crew_options_metrics(),
options_cluster = crew.cluster::crew_options_pbs(),
verbose = NULL,
command_submit = NULL,
command_terminate = NULL,
command_delete = NULL,
script_directory = NULL,
script_lines = NULL,
pbs_cwd = NULL,
pbs_log_output = NULL,
pbs_log_error = NULL,
pbs_log_join = NULL,
pbs_memory_gigabytes_required = NULL,
pbs_cores = NULL,
pbs_walltime_hours = NULL
)
Arguments
name |
Character string, name of the launcher. If the name is
|
workers |
Maximum number of workers to run concurrently
when auto-scaling, excluding task retries and manual calls to |
host |
IP address of the |
port |
TCP port to listen for the workers. If |
tls |
A TLS configuration object from |
tls_enable |
Deprecated on 2023-09-15 in version 0.4.1.
Use argument |
tls_config |
Deprecated on 2023-09-15 in version 0.4.1.
Use argument |
serialization |
Either |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete. In certain cases, exponential
backoff is used with this argument passed to |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
seconds_exit |
Deprecated on 2023-09-21 in version 0.1.2.9000. No longer necessary. |
retry_tasks |
Deprecated on 2025-01-13 ( |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
|
reset_packages |
|
reset_options |
|
garbage_collection |
|
crashes_error |
Deprecated on 2025-01-13 ( |
r_arguments |
Optional character vector of command line arguments
to pass to |
crashes_max |
In rare cases, a worker may exit unexpectedly
before it completes its current task. If this happens,
|
backup |
An optional Limitations of |
options_metrics |
Either |
options_cluster |
An options list from
|
verbose |
Deprecated. Use |
command_submit |
Deprecated. Use |
command_terminate |
Deprecated. Use |
command_delete |
Deprecated on 2024-01-08 (version 0.1.4.9001).
Use |
script_directory |
Deprecated. Use |
script_lines |
Deprecated. Use |
pbs_cwd |
Deprecated. Use |
pbs_log_output |
Deprecated. Use |
pbs_log_error |
Deprecated. Use |
pbs_log_join |
Deprecated. Use |
pbs_memory_gigabytes_required |
Deprecated.
Use |
pbs_cores |
Deprecated. Use |
pbs_walltime_hours |
Deprecated. Use |
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other pbs:
crew_class_launcher_pbs
,
crew_launcher_pbs()
,
crew_options_pbs()
Examples
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
controller <- crew_controller_pbs()
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()$result
controller$terminate()
}
Create a controller with a
Sun Grid Engine (SGE) launcher.
Description
Create an R6
object to submit tasks and
launch workers on Sun Grid Engine (SGE) workers.
Usage
crew_controller_sge(
name = NULL,
workers = 1L,
host = NULL,
port = NULL,
tls = crew::crew_tls(mode = "automatic"),
tls_enable = NULL,
tls_config = NULL,
serialization = NULL,
seconds_interval = 0.25,
seconds_timeout = 60,
seconds_launch = 86400,
seconds_idle = 300,
seconds_wall = Inf,
seconds_exit = NULL,
retry_tasks = NULL,
tasks_max = Inf,
tasks_timers = 0L,
reset_globals = TRUE,
reset_packages = FALSE,
reset_options = FALSE,
garbage_collection = FALSE,
crashes_error = NULL,
r_arguments = c("--no-save", "--no-restore"),
crashes_max = 5L,
backup = NULL,
options_metrics = crew::crew_options_metrics(),
options_cluster = crew.cluster::crew_options_sge(),
verbose = NULL,
command_submit = NULL,
command_terminate = NULL,
command_delete = NULL,
script_directory = NULL,
script_lines = NULL,
sge_cwd = NULL,
sge_envvars = NULL,
sge_log_output = NULL,
sge_log_error = NULL,
sge_log_join = NULL,
sge_memory_gigabytes_limit = NULL,
sge_memory_gigabytes_required = NULL,
sge_cores = NULL,
sge_gpu = NULL
)
Arguments
name |
Character string, name of the launcher. If the name is
|
workers |
Maximum number of workers to run concurrently
when auto-scaling, excluding task retries and manual calls to |
host |
IP address of the |
port |
TCP port to listen for the workers. If |
tls |
A TLS configuration object from |
tls_enable |
Deprecated on 2023-09-15 in version 0.4.1.
Use argument |
tls_config |
Deprecated on 2023-09-15 in version 0.4.1.
Use argument |
serialization |
Either |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete. In certain cases, exponential
backoff is used with this argument passed to |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
seconds_exit |
Deprecated on 2023-09-21 in version 0.1.2.9000. No longer necessary. |
retry_tasks |
Deprecated on 2025-01-13 ( |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
|
reset_packages |
|
reset_options |
|
garbage_collection |
|
crashes_error |
Deprecated on 2025-01-13 ( |
r_arguments |
Optional character vector of command line arguments
to pass to |
crashes_max |
In rare cases, a worker may exit unexpectedly
before it completes its current task. If this happens,
|
backup |
An optional Limitations of |
options_metrics |
Either |
options_cluster |
An options list from
|
verbose |
Deprecated. Use |
command_submit |
Deprecated. Use |
command_terminate |
Deprecated. Use |
command_delete |
Deprecated on 2024-01-08 (version 0.1.4.9001).
Use |
script_directory |
Deprecated. Use |
script_lines |
Deprecated. Use |
sge_cwd |
Deprecated. Use |
sge_envvars |
Deprecated. Use |
sge_log_output |
Deprecated. Use |
sge_log_error |
Deprecated. Use |
sge_log_join |
Deprecated. Use |
sge_memory_gigabytes_limit |
Deprecated.
Use |
sge_memory_gigabytes_required |
Deprecated.
Use |
sge_cores |
Deprecated. Use |
sge_gpu |
Deprecated. Use |
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other sge:
crew_class_launcher_sge
,
crew_class_monitor_sge
,
crew_launcher_sge()
,
crew_monitor_sge()
,
crew_options_sge()
Examples
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
controller <- crew_controller_sge()
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()$result
controller$terminate()
}
Create a controller with a
SLURM launcher.
Description
Create an R6
object to submit tasks and
launch workers on SLURM workers.
Usage
crew_controller_slurm(
name = NULL,
workers = 1L,
host = NULL,
port = NULL,
tls = crew::crew_tls(mode = "automatic"),
tls_enable = NULL,
tls_config = NULL,
serialization = NULL,
seconds_interval = 0.25,
seconds_timeout = 60,
seconds_launch = 86400,
seconds_idle = 300,
seconds_wall = Inf,
seconds_exit = NULL,
retry_tasks = NULL,
tasks_max = Inf,
tasks_timers = 0L,
reset_globals = TRUE,
reset_packages = FALSE,
reset_options = FALSE,
garbage_collection = FALSE,
crashes_error = NULL,
r_arguments = c("--no-save", "--no-restore"),
crashes_max = 5L,
backup = NULL,
options_metrics = crew::crew_options_metrics(),
options_cluster = crew.cluster::crew_options_slurm(),
verbose = NULL,
command_submit = NULL,
command_terminate = NULL,
command_delete = NULL,
script_directory = NULL,
script_lines = NULL,
slurm_log_output = NULL,
slurm_log_error = NULL,
slurm_memory_gigabytes_required = NULL,
slurm_memory_gigabytes_per_cpu = NULL,
slurm_cpus_per_task = NULL,
slurm_time_minutes = NULL,
slurm_partition = NULL
)
Arguments
name |
Character string, name of the launcher. If the name is
|
workers |
Maximum number of workers to run concurrently
when auto-scaling, excluding task retries and manual calls to |
host |
IP address of the |
port |
TCP port to listen for the workers. If |
tls |
A TLS configuration object from |
tls_enable |
Deprecated on 2023-09-15 in version 0.4.1.
Use argument |
tls_config |
Deprecated on 2023-09-15 in version 0.4.1.
Use argument |
serialization |
Either |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete. In certain cases, exponential
backoff is used with this argument passed to |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
seconds_exit |
Deprecated on 2023-09-21 in version 0.1.2.9000. No longer necessary. |
retry_tasks |
Deprecated on 2025-01-13 ( |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
|
reset_packages |
|
reset_options |
|
garbage_collection |
|
crashes_error |
Deprecated on 2025-01-13 ( |
r_arguments |
Optional character vector of command line arguments
to pass to |
crashes_max |
In rare cases, a worker may exit unexpectedly
before it completes its current task. If this happens,
|
backup |
An optional Limitations of |
options_metrics |
Either |
options_cluster |
An options list from
|
verbose |
Deprecated. Use |
command_submit |
Deprecated. Use |
command_terminate |
Deprecated. Use |
command_delete |
Deprecated on 2024-01-08 (version 0.1.4.9001).
Use |
script_directory |
Deprecated. Use |
script_lines |
Deprecated. Use |
slurm_log_output |
Deprecated. Use |
slurm_log_error |
Deprecated. Use |
slurm_memory_gigabytes_required |
Deprecated.
Use |
slurm_memory_gigabytes_per_cpu |
Deprecated.
Use |
slurm_cpus_per_task |
Deprecated. Use |
slurm_time_minutes |
Deprecated. Use |
slurm_partition |
Deprecated. Use |
Details
WARNING: the crew.cluster
SLURM plugin is experimental
and has not actually been tested on a SLURM cluster. Please proceed
with caution and report bugs to
https://github.com/wlandau/crew.cluster.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other slurm:
crew_class_launcher_slurm
,
crew_class_monitor_slurm
,
crew_launcher_slurm()
,
crew_monitor_slurm()
,
crew_options_slurm()
Examples
if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
controller <- crew_controller_slurm()
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()$result
controller$terminate()
}
Create an abstract cluster
launcher object.
Description
Create an R6
abstract cluster launcher object.
Usage
crew_launcher_cluster(
name = NULL,
workers = 1L,
seconds_interval = 0.5,
seconds_timeout = 60,
seconds_launch = 86400,
seconds_idle = 300,
seconds_wall = Inf,
tasks_max = Inf,
tasks_timers = 0L,
reset_globals = NULL,
reset_packages = NULL,
reset_options = NULL,
garbage_collection = NULL,
crashes_error = NULL,
tls = crew::crew_tls(mode = "automatic"),
r_arguments = c("--no-save", "--no-restore"),
options_metrics = crew::crew_options_metrics(),
options_cluster = crew.cluster::crew_options_cluster(),
verbose = NULL,
command_submit = NULL,
command_terminate = NULL,
command_delete = NULL,
script_directory = NULL,
script_lines = NULL
)
Arguments
name |
Character string, name of the launcher. If the name is
|
workers |
Maximum number of workers to run concurrently
when auto-scaling, excluding task retries and manual calls to |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete. In certain cases, exponential
backoff is used with this argument passed to |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
Deprecated on 2025-05-30 ( |
reset_packages |
Deprecated on 2025-05-30 ( |
reset_options |
Deprecated on 2025-05-30 ( |
garbage_collection |
Deprecated on 2025-05-30
( |
crashes_error |
Deprecated on 2025-01-13 ( |
tls |
A TLS configuration object from |
r_arguments |
Optional character vector of command line arguments
to pass to |
options_metrics |
Either |
options_cluster |
List of options from a |
verbose |
Deprecated. Use |
command_submit |
Deprecated. Use |
command_terminate |
Deprecated. Use |
command_delete |
Deprecated on 2024-01-08 (version 0.1.4.9001).
Use |
script_directory |
Deprecated. Use |
script_lines |
Deprecated. Use |
Details
This abstract class is used to develop specific launcher classes for specific computing platforms.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other cluster:
crew_class_launcher_cluster
,
crew_class_monitor_cluster
,
crew_monitor_cluster()
,
crew_options_cluster()
Create a launcher with
LSF workers.
Description
Create an R6
object to launch and maintain
workers as LSF jobs.
Usage
crew_launcher_lsf(
name = NULL,
workers = 1L,
seconds_interval = 0.5,
seconds_timeout = 60,
seconds_launch = 86400,
seconds_idle = 300,
seconds_wall = Inf,
tasks_max = Inf,
tasks_timers = 0L,
reset_globals = NULL,
reset_packages = NULL,
reset_options = NULL,
garbage_collection = NULL,
crashes_error = NULL,
tls = crew::crew_tls(mode = "automatic"),
r_arguments = c("--no-save", "--no-restore"),
options_metrics = crew::crew_options_metrics(),
options_cluster = crew.cluster::crew_options_lsf(),
verbose = NULL,
command_submit = NULL,
command_terminate = NULL,
command_delete = NULL,
script_directory = NULL,
script_lines = NULL,
lsf_cwd = NULL,
lsf_log_output = NULL,
lsf_log_error = NULL,
lsf_memory_gigabytes_limit = NULL,
lsf_memory_gigabytes_required = NULL,
lsf_cores = NULL
)
Arguments
name |
Character string, name of the launcher. If the name is
|
workers |
Maximum number of workers to run concurrently
when auto-scaling, excluding task retries and manual calls to |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete. In certain cases, exponential
backoff is used with this argument passed to |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
Deprecated on 2025-05-30 ( |
reset_packages |
Deprecated on 2025-05-30 ( |
reset_options |
Deprecated on 2025-05-30 ( |
garbage_collection |
Deprecated on 2025-05-30
( |
crashes_error |
Deprecated on 2025-01-13 ( |
tls |
A TLS configuration object from |
r_arguments |
Optional character vector of command line arguments
to pass to |
options_metrics |
Either |
options_cluster |
An options list from
|
verbose |
Deprecated. Use |
command_submit |
Deprecated. Use |
command_terminate |
Deprecated. Use |
command_delete |
Deprecated on 2024-01-08 (version 0.1.4.9001).
Use |
script_directory |
Deprecated. Use |
script_lines |
Deprecated. Use |
lsf_cwd |
Deprecated. Use |
lsf_log_output |
Deprecated. Use |
lsf_log_error |
Deprecated. Use |
lsf_memory_gigabytes_limit |
Deprecated.
Use |
lsf_memory_gigabytes_required |
Deprecated.
Use |
lsf_cores |
Deprecated. Use |
Details
WARNING: the crew.cluster
LSF plugin is experimental.
Please proceed with caution and report bugs to
https://github.com/wlandau/crew.cluster.
To launch a LSF worker, this launcher
creates a temporary job script with a call to crew::crew_worker()
and submits it as an LSF job with sbatch
. To see most of the lines
of the job script in advance, use the script()
method of the launcher.
It has all the lines except for the job name and the
call to crew::crew_worker()
, both of
which will be inserted at the last minute when it is time
to actually launch a worker.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other lsf:
crew_class_launcher_lsf
,
crew_controller_lsf()
,
crew_options_lsf()
Create a launcher with
PBS or TORQUE workers.
Description
Create an R6
object to launch and maintain
workers as jobs on a PBS or TORQUE cluster.
Usage
crew_launcher_pbs(
name = NULL,
workers = 1L,
seconds_interval = 0.5,
seconds_timeout = 60,
seconds_launch = 86400,
seconds_idle = 300,
seconds_wall = Inf,
tasks_max = Inf,
tasks_timers = 0L,
reset_globals = NULL,
reset_packages = NULL,
reset_options = NULL,
garbage_collection = NULL,
crashes_error = NULL,
tls = crew::crew_tls(mode = "automatic"),
r_arguments = c("--no-save", "--no-restore"),
options_metrics = crew::crew_options_metrics(),
options_cluster = crew.cluster::crew_options_pbs(),
verbose = NULL,
command_submit = NULL,
command_terminate = NULL,
command_delete = NULL,
script_directory = NULL,
script_lines = NULL,
pbs_cwd = NULL,
pbs_log_output = NULL,
pbs_log_error = NULL,
pbs_log_join = NULL,
pbs_memory_gigabytes_required = NULL,
pbs_cores = NULL,
pbs_walltime_hours = NULL
)
Arguments
name |
Character string, name of the launcher. If the name is
|
workers |
Maximum number of workers to run concurrently
when auto-scaling, excluding task retries and manual calls to |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete. In certain cases, exponential
backoff is used with this argument passed to |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
Deprecated on 2025-05-30 ( |
reset_packages |
Deprecated on 2025-05-30 ( |
reset_options |
Deprecated on 2025-05-30 ( |
garbage_collection |
Deprecated on 2025-05-30
( |
crashes_error |
Deprecated on 2025-01-13 ( |
tls |
A TLS configuration object from |
r_arguments |
Optional character vector of command line arguments
to pass to |
options_metrics |
Either |
options_cluster |
An options list from
|
verbose |
Deprecated. Use |
command_submit |
Deprecated. Use |
command_terminate |
Deprecated. Use |
command_delete |
Deprecated on 2024-01-08 (version 0.1.4.9001).
Use |
script_directory |
Deprecated. Use |
script_lines |
Deprecated. Use |
pbs_cwd |
Deprecated. Use |
pbs_log_output |
Deprecated. Use |
pbs_log_error |
Deprecated. Use |
pbs_log_join |
Deprecated. Use |
pbs_memory_gigabytes_required |
Deprecated.
Use |
pbs_cores |
Deprecated. Use |
pbs_walltime_hours |
Deprecated. Use |
Details
WARNING: the crew.cluster
PBS plugin is experimental
and has not actually been tested on a PBS cluster. Please proceed
with caution and report bugs to
https://github.com/wlandau/crew.cluster.
To launch a PBS/TORQUE worker, this launcher
creates a temporary job script with a call to crew::crew_worker()
and submits it as an PBS job with qsub
. To see most of the lines
of the job script in advance, use the script()
method of the launcher.
It has all the lines except for the job name and the
call to crew::crew_worker()
, both of
which will be inserted at the last minute when it is time
to actually launch a worker.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other pbs:
crew_class_launcher_pbs
,
crew_controller_pbs()
,
crew_options_pbs()
Create a launcher with
Sun Grid Engine (SGE) workers.
Description
Create an R6
object to launch and maintain
workers as Sun Grid Engine (SGE) jobs.
Usage
crew_launcher_sge(
name = NULL,
workers = 1L,
seconds_interval = 0.5,
seconds_timeout = 60,
seconds_launch = 86400,
seconds_idle = 300,
seconds_wall = Inf,
tasks_max = Inf,
tasks_timers = 0L,
reset_globals = NULL,
reset_packages = NULL,
reset_options = NULL,
garbage_collection = NULL,
crashes_error = NULL,
tls = crew::crew_tls(mode = "automatic"),
r_arguments = c("--no-save", "--no-restore"),
options_metrics = crew::crew_options_metrics(),
options_cluster = crew.cluster::crew_options_sge(),
verbose = NULL,
command_submit = NULL,
command_terminate = NULL,
command_delete = NULL,
script_directory = NULL,
script_lines = NULL,
sge_cwd = NULL,
sge_envvars = NULL,
sge_log_output = NULL,
sge_log_error = NULL,
sge_log_join = NULL,
sge_memory_gigabytes_limit = NULL,
sge_memory_gigabytes_required = NULL,
sge_cores = NULL,
sge_gpu = NULL
)
Arguments
name |
Character string, name of the launcher. If the name is
|
workers |
Maximum number of workers to run concurrently
when auto-scaling, excluding task retries and manual calls to |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete. In certain cases, exponential
backoff is used with this argument passed to |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
Deprecated on 2025-05-30 ( |
reset_packages |
Deprecated on 2025-05-30 ( |
reset_options |
Deprecated on 2025-05-30 ( |
garbage_collection |
Deprecated on 2025-05-30
( |
crashes_error |
Deprecated on 2025-01-13 ( |
tls |
A TLS configuration object from |
r_arguments |
Optional character vector of command line arguments
to pass to |
options_metrics |
Either |
options_cluster |
An options list from
|
verbose |
Deprecated. Use |
command_submit |
Deprecated. Use |
command_terminate |
Deprecated. Use |
command_delete |
Deprecated on 2024-01-08 (version 0.1.4.9001).
Use |
script_directory |
Deprecated. Use |
script_lines |
Deprecated. Use |
sge_cwd |
Deprecated. Use |
sge_envvars |
Deprecated. Use |
sge_log_output |
Deprecated. Use |
sge_log_error |
Deprecated. Use |
sge_log_join |
Deprecated. Use |
sge_memory_gigabytes_limit |
Deprecated.
Use |
sge_memory_gigabytes_required |
Deprecated.
Use |
sge_cores |
Deprecated. Use |
sge_gpu |
Deprecated. Use |
Details
To launch a Sun Grid Engine (SGE) worker, this launcher
creates a temporary job script with a call to crew::crew_worker()
and submits it as an SGE job with qsub
. To see most of the lines
of the job script in advance, use the script()
method of the launcher.
It has all the lines except for the job name and the
call to crew::crew_worker()
, both of
which will be inserted at the last minute when it is time
to actually launch a worker.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other sge:
crew_class_launcher_sge
,
crew_class_monitor_sge
,
crew_controller_sge()
,
crew_monitor_sge()
,
crew_options_sge()
Create a launcher with
SLURM workers.
Description
Create an R6
object to launch and maintain
workers as SLURM jobs.
Usage
crew_launcher_slurm(
name = NULL,
workers = 1L,
seconds_interval = 0.5,
seconds_timeout = 60,
seconds_launch = 86400,
seconds_idle = 300,
seconds_wall = Inf,
tasks_max = Inf,
tasks_timers = 0L,
reset_globals = NULL,
reset_packages = NULL,
reset_options = NULL,
garbage_collection = NULL,
crashes_error = NULL,
tls = crew::crew_tls(mode = "automatic"),
r_arguments = c("--no-save", "--no-restore"),
options_metrics = crew::crew_options_metrics(),
options_cluster = crew.cluster::crew_options_slurm(),
verbose = NULL,
command_submit = NULL,
command_terminate = NULL,
command_delete = NULL,
script_directory = NULL,
script_lines = NULL,
slurm_log_output = NULL,
slurm_log_error = NULL,
slurm_memory_gigabytes_required = NULL,
slurm_memory_gigabytes_per_cpu = NULL,
slurm_cpus_per_task = NULL,
slurm_time_minutes = NULL,
slurm_partition = NULL
)
Arguments
name |
Character string, name of the launcher. If the name is
|
workers |
Maximum number of workers to run concurrently
when auto-scaling, excluding task retries and manual calls to |
seconds_interval |
Number of seconds between
polling intervals waiting for certain internal
synchronous operations to complete. In certain cases, exponential
backoff is used with this argument passed to |
seconds_timeout |
Number of seconds until timing
out while waiting for certain synchronous operations to complete,
such as checking |
seconds_launch |
Seconds of startup time to allow.
A worker is unconditionally assumed to be alive
from the moment of its launch until |
seconds_idle |
Maximum number of seconds that a worker can idle
since the completion of the last task. If exceeded, the worker exits.
But the timer does not launch until |
seconds_wall |
Soft wall time in seconds.
The timer does not launch until |
tasks_max |
Maximum number of tasks that a worker will do before
exiting. See the |
tasks_timers |
Number of tasks to do before activating
the timers for |
reset_globals |
Deprecated on 2025-05-30 ( |
reset_packages |
Deprecated on 2025-05-30 ( |
reset_options |
Deprecated on 2025-05-30 ( |
garbage_collection |
Deprecated on 2025-05-30
( |
crashes_error |
Deprecated on 2025-01-13 ( |
tls |
A TLS configuration object from |
r_arguments |
Optional character vector of command line arguments
to pass to |
options_metrics |
Either |
options_cluster |
An options list from
|
verbose |
Deprecated. Use |
command_submit |
Deprecated. Use |
command_terminate |
Deprecated. Use |
command_delete |
Deprecated on 2024-01-08 (version 0.1.4.9001).
Use |
script_directory |
Deprecated. Use |
script_lines |
Deprecated. Use |
slurm_log_output |
Deprecated. Use |
slurm_log_error |
Deprecated. Use |
slurm_memory_gigabytes_required |
Deprecated.
Use |
slurm_memory_gigabytes_per_cpu |
Deprecated.
Use |
slurm_cpus_per_task |
Deprecated. Use |
slurm_time_minutes |
Deprecated. Use |
slurm_partition |
Deprecated. Use |
Details
WARNING: the crew.cluster
SLURM plugin is experimental
and has not actually been tested on a SLURM cluster. Please proceed
with caution and report bugs to
https://github.com/wlandau/crew.cluster.
To launch a SLURM worker, this launcher
creates a temporary job script with a call to crew::crew_worker()
and submits it as an SLURM job with sbatch
. To see most of the lines
of the job script in advance, use the script()
method of the launcher.
It has all the lines except for the job name and the
call to crew::crew_worker()
, both of
which will be inserted at the last minute when it is time
to actually launch a worker.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other slurm:
crew_class_launcher_slurm
,
crew_class_monitor_slurm
,
crew_controller_slurm()
,
crew_monitor_slurm()
,
crew_options_slurm()
Create an abstract
cluster monitor object.
Description
Create an abstract cluster monitor R6
object.
Usage
crew_monitor_cluster(
verbose = TRUE,
command_list = as.character(Sys.which("qstat")),
command_terminate = as.character(Sys.which("qdel"))
)
Arguments
verbose |
Deprecated. Use |
command_list |
Character of length 1, file path to the executable to list jobs. |
command_terminate |
Deprecated. Use |
See Also
Other cluster:
crew_class_launcher_cluster
,
crew_class_monitor_cluster
,
crew_launcher_cluster()
,
crew_options_cluster()
Create a SGE monitor object.
Description
Create an R6
object to monitor SGE cluster jobs.
Usage
crew_monitor_sge(
verbose = TRUE,
command_list = as.character(Sys.which("qstat")),
command_terminate = as.character(Sys.which("qdel"))
)
Arguments
verbose |
Deprecated. Use |
command_list |
Character of length 1, file path to the executable to list jobs. |
command_terminate |
Deprecated. Use |
See Also
Other sge:
crew_class_launcher_sge
,
crew_class_monitor_sge
,
crew_controller_sge()
,
crew_launcher_sge()
,
crew_options_sge()
Create a SLURM monitor object.
Description
Create an R6
object to monitor SLURM cluster jobs.
Usage
crew_monitor_slurm(
verbose = TRUE,
command_list = as.character(Sys.which("squeue")),
command_terminate = as.character(Sys.which("scancel"))
)
Arguments
verbose |
Deprecated. Use |
command_list |
Character of length 1, file path to the executable to list jobs. |
command_terminate |
Deprecated. Use |
See Also
Other slurm:
crew_class_launcher_slurm
,
crew_class_monitor_slurm
,
crew_controller_slurm()
,
crew_launcher_slurm()
,
crew_options_slurm()
Common abstract cluster options.
Description
Common abstract cluster options.
Usage
crew_options_cluster(
verbose = FALSE,
command_submit = as.character(Sys.which("qsub")),
command_terminate = as.character(Sys.which("qdel")),
script_directory = tempdir(),
script_lines = character(0L)
)
Arguments
verbose |
Logical, whether to see console output and error messages when submitting worker. |
command_submit |
Character of length 1, file path to the executable to submit a worker job. |
command_terminate |
Character of length 1,
file path to the executable to terminate a worker job.
Set to |
script_directory |
Character of length 1, directory path to the
job scripts. Just before each job submission, a job script
is created in this folder. Script base names are unique to each
launcher and worker, and the launcher deletes the script when the
worker is manually terminated. |
script_lines |
Optional character vector of additional lines to be
added to the job script just after the more common flags.
An example would be |
Value
A classed list of options.
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other cluster:
crew_class_launcher_cluster
,
crew_class_monitor_cluster
,
crew_launcher_cluster()
,
crew_monitor_cluster()
Examples
crew_options_cluster()
LSF options.
Description
Set options for LSF job management.
Usage
crew_options_lsf(
verbose = FALSE,
command_submit = as.character(Sys.which("bsub")),
command_terminate = as.character(Sys.which("bkill")),
script_directory = tempdir(),
script_lines = character(0L),
cwd = getwd(),
log_output = "/dev/null",
log_error = "/dev/null",
memory_gigabytes_limit = NULL,
memory_gigabytes_required = NULL,
cores = NULL
)
Arguments
verbose |
Logical, whether to see console output and error messages when submitting worker. |
command_submit |
Character of length 1, file path to the executable to submit a worker job. |
command_terminate |
Character of length 1,
file path to the executable to terminate a worker job.
Set to |
script_directory |
Character of length 1, directory path to the
job scripts. Just before each job submission, a job script
is created in this folder. Script base names are unique to each
launcher and worker, and the launcher deletes the script when the
worker is manually terminated. |
script_lines |
Optional character vector of additional lines to be
added to the job script just after the more common flags.
An example would be |
cwd |
Character of length 1, directory to
launch the worker from (as opposed to
the system default). |
log_output |
Character of length 1, file pattern to control
the locations of the LSF worker log files. By default, both standard
output and standard error go to the same file.
|
log_error |
Character of length 1, file pattern for standard
error. |
memory_gigabytes_limit |
Positive numeric scalar, memory
limit in gigabytes of the worker.
|
memory_gigabytes_required |
Positive numeric scalar,
memory requirement in gigabytes.
|
cores |
Optional positive integer scalar,
number of CPU cores for the worker.
|
Value
A classed list of options.
Retryable options
Retryable options are deprecated in crew.cluster
as of
2025-01-27 (version 0.3.4
).
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other lsf:
crew_class_launcher_lsf
,
crew_controller_lsf()
,
crew_launcher_lsf()
Examples
crew_options_lsf()
PBS options.
Description
Set options for PBS job management.
Usage
crew_options_pbs(
verbose = FALSE,
command_submit = as.character(Sys.which("qsub")),
command_terminate = as.character(Sys.which("qdel")),
script_directory = tempdir(),
script_lines = character(0L),
cwd = TRUE,
log_output = "/dev/null",
log_error = NULL,
log_join = TRUE,
memory_gigabytes_required = NULL,
cores = NULL,
walltime_hours = 12
)
Arguments
verbose |
Logical, whether to see console output and error messages when submitting worker. |
command_submit |
Character of length 1, file path to the executable to submit a worker job. |
command_terminate |
Character of length 1,
file path to the executable to terminate a worker job.
Set to |
script_directory |
Character of length 1, directory path to the
job scripts. Just before each job submission, a job script
is created in this folder. Script base names are unique to each
launcher and worker, and the launcher deletes the script when the
worker is manually terminated. |
script_lines |
Optional character vector of additional lines to be
added to the job script just after the more common flags.
An example would be |
cwd |
Logical of length 1, whether to set the working directory
of the worker to the working directory it was launched from.
|
log_output |
Character of length 1, file or directory path to PBS
worker log files for standard output.
|
log_error |
Character of length 1, file or directory path to PBS
worker log files for standard error.
|
log_join |
Logical, whether to join the stdout and stderr log
files together into one file. |
memory_gigabytes_required |
Optional positive numeric scalar,
gigabytes of memory required to run the worker.
|
cores |
Optional positive integer scalar,
number of cores for the worker
("slots" in PBS lingo).
|
walltime_hours |
Numeric scalar,
hours of wall time
to request for the worker. |
Value
A classed list of options.
Retryable options
Retryable options are deprecated in crew.cluster
as of
2025-01-27 (version 0.3.4
).
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other pbs:
crew_class_launcher_pbs
,
crew_controller_pbs()
,
crew_launcher_pbs()
Examples
crew_options_pbs()
SGE options.
Description
Set options for SGE job management.
Usage
crew_options_sge(
verbose = FALSE,
command_submit = as.character(Sys.which("qsub")),
command_terminate = as.character(Sys.which("qdel")),
script_directory = tempdir(),
script_lines = character(0L),
cwd = TRUE,
envvars = FALSE,
log_output = "/dev/null",
log_error = NULL,
log_join = TRUE,
memory_gigabytes_limit = NULL,
memory_gigabytes_required = NULL,
cores = NULL,
gpu = NULL
)
Arguments
verbose |
Logical, whether to see console output and error messages when submitting worker. |
command_submit |
Character of length 1, file path to the executable to submit a worker job. |
command_terminate |
Character of length 1,
file path to the executable to terminate a worker job.
Set to |
script_directory |
Character of length 1, directory path to the
job scripts. Just before each job submission, a job script
is created in this folder. Script base names are unique to each
launcher and worker, and the launcher deletes the script when the
worker is manually terminated. |
script_lines |
Optional character vector of additional lines to be
added to the job script just after the more common flags.
An example would be |
cwd |
Logical of length 1, whether to
launch the worker from the current working directory (as opposed to
the user home directory). |
envvars |
Logical of length 1, whether to forward the environment
variables of the current session to the SGE worker. |
log_output |
Character of length 1, file or directory path to SGE
worker log files for standard output.
|
log_error |
Character of length 1, file or directory path to SGE
worker log files for standard error.
|
log_join |
Logical, whether to join the stdout and stderr log
files together into one file. |
memory_gigabytes_limit |
Optional numeric scalar,
maximum number of gigabytes of memory a worker is allowed to
consume. If the worker consumes more than this level of memory, then
SGE will terminate it. |
memory_gigabytes_required |
Optional positive numeric scalar,
gigabytes of memory required to run the worker.
|
cores |
Optional positive integer scalar,
number of cores per worker ("slots" in SGE lingo).
|
gpu |
Optional integer scalar,
number of GPUs to
request for the worker. |
Value
A classed list of options.
Retryable options
Retryable options are deprecated in crew.cluster
as of
2025-01-27 (version 0.3.4
).
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other sge:
crew_class_launcher_sge
,
crew_class_monitor_sge
,
crew_controller_sge()
,
crew_launcher_sge()
,
crew_monitor_sge()
Examples
crew_options_sge()
SLURM options.
Description
Set options for SLURM job management.
Usage
crew_options_slurm(
verbose = FALSE,
command_submit = as.character(Sys.which("sbatch")),
command_terminate = as.character(Sys.which("scancel")),
script_directory = tempdir(),
script_lines = character(0L),
log_output = "/dev/null",
log_error = "/dev/null",
memory_gigabytes_required = NULL,
memory_gigabytes_per_cpu = NULL,
cpus_per_task = NULL,
time_minutes = NULL,
partition = NULL,
n_tasks = 1
)
Arguments
verbose |
Logical, whether to see console output and error messages when submitting worker. |
command_submit |
Character of length 1, file path to the executable to submit a worker job. |
command_terminate |
Character of length 1,
file path to the executable to terminate a worker job.
Set to |
script_directory |
Character of length 1, directory path to the
job scripts. Just before each job submission, a job script
is created in this folder. Script base names are unique to each
launcher and worker, and the launcher deletes the script when the
worker is manually terminated. |
script_lines |
Optional character vector of additional lines to be
added to the job script just after the more common flags.
An example would be |
log_output |
Character of length 1, file pattern to control
the locations of the SLURM worker log files. By default, both standard
output and standard error go to the same file.
|
log_error |
Character of length 1, file pattern for standard
error. |
memory_gigabytes_required |
Positive numeric scalar,
total number of gigabytes of memory required per node.
|
memory_gigabytes_per_cpu |
Positive numeric scalar,
gigabytes of
memory required per CPU.
|
cpus_per_task |
Optional positive integer scalar,
number of CPUs for the worker.
|
time_minutes |
Numeric scalar,
number of minutes to
designate as the wall time of |
partition |
Character string,
name of the SLURM partition to
create workers on. |
n_tasks |
Numeric scalar, number of SLURM tasks to run
within the job.
|
Value
A classed list of options.
Retryable options
Retryable options are deprecated in crew.cluster
as of
2025-01-27 (version 0.3.4
).
Attribution
The template files at
https://github.com/mschubert/clustermq/tree/master/inst
informed the development of the crew
launcher plugins in
crew.cluster
, and we would like to thank
Michael Schubert for developing clustermq
and releasing it under
the permissive Apache License 2.0.
See the NOTICE
and README.md
files in the crew.cluster
source code for additional attribution.
See Also
Other slurm:
crew_class_launcher_slurm
,
crew_class_monitor_slurm
,
crew_controller_slurm()
,
crew_launcher_slurm()
,
crew_monitor_slurm()
Examples
crew_options_slurm()