compare runs

Description

compare runs

Usage

.view_runs_diff(
  runs = "1:2",
  output_dir = file.path(tempdir(), paste(ids, collapse = "-")),
  template = system.file("templates/compare-runs.qmd", package = "guildai"),
  viewer = getOption("guildai.viewer"),
  ...
)

Arguments

Value

path to the generated html, invisibly

guild_api_runs_cli Show runs as JSON.

Description

IMPORTANT: This command is experimental and subject to change without notice.

Usage

guild_api_runs_cli(
  ...,
  format = NA,
  filter = NULL,
  operation = NULL,
  label = NULL,
  unlabeled = NA,
  tag = NULL,
  comment = NULL,
  marked = NA,
  unmarked = NA,
  started = NULL,
  digest = NULL,
  running = NA,
  completed = NA,
  error = NA,
  terminated = NA,
  pending = NA,
  staged = NA,
  api_version = NULL,
  deleted = NA,
  include_batch = NA
)

Arguments

Copy run files into the current project working directory

Description

Copy run files into the current project working directory

Usage

guild_merge(run = NULL, ...)

Arguments

Value

NULL, invisibly. This function is called for its side effect.

Examples

## Not run: 
guild_merge("--help")
runs_scalars() %>%
  dplyr::slice_max("epoch_acc") %>%
  guild_merge(I("--yes --replace"))

## End(Not run)

guild_merge_cli Copy run files to a project directory.

Description

By default, Guild copies run files into the current directory. To copy files to a different directory, use ⁠target_dir DIR⁠.

Usage

guild_merge_cli(
  ...,
  filter = NULL,
  operation = NULL,
  label = NULL,
  unlabeled = NA,
  tag = NULL,
  comment = NULL,
  marked = NA,
  unmarked = NA,
  started = NULL,
  digest = NULL,
  running = NA,
  completed = NA,
  error = NA,
  terminated = NA,
  pending = NA,
  staged = NA,
  target_dir = NULL,
  sourcecode = NA,
  all = NA,
  skip_sourcecode = NA,
  skip_deps = NA,
  exclude = NULL,
  no_summary = NA,
  summary_name = NULL,
  preview = NA,
  replace = NA,
  no_replace = NA
)

Arguments

Details

Guild checks that the run originated from the current directory before copying files. If the run is associated with a project from a different directory, or is from a package, Guild exits with an error message. In this case, use target_dir to override the check with an explicit path.

The command fails if any file would be replaced, unless a) the replace option is specified or b) the replaced file is committed to the project VCS and unchanged. To prevent replacement even when a file is committed to VCS and unchanged, specify no_replace.

Launch a guild run

Description

Launch a guild run

Usage

guild_run(
  opspec = "train.R",
  flags = NULL,
  ...,
  echo = TRUE,
  as_job = getOption("guildai.run_as_job", TRUE)
)

Arguments

Value

NULL, invisibly. This function is called for its side effect.

guild_run_cli Run an operation.

Description

By default Guild tries to run OPERATION for the default model defined in the current project.

Usage

guild_run_cli(
  ...,
  label = NULL,
  tag = NULL,
  comment = NULL,
  run_dir = NULL,
  stage = NA,
  start = NULL,
  restart = NULL,
  proto = NULL,
  force_sourcecode = NA,
  gpus = NULL,
  no_gpus = NA,
  batch_label = NULL,
  batch_tag = NULL,
  batch_comment = NULL,
  optimizer = NULL,
  optimize = NA,
  minimize = NULL,
  maximize = NULL,
  opt_flag = NULL,
  max_trials = NULL,
  trials = NULL,
  stage_trials = NA,
  remote = NULL,
  force_flags = NA,
  force_deps = NA,
  stop_after = NULL,
  fail_on_trial_error = NA,
  needed = NA,
  background = NA,
  pidfile = NULL,
  no_wait = NA,
  save_trials = NULL,
  keep_run = NA,
  keep_batch = NA,
  dep = NULL,
  quiet = NA,
  print_cmd = NA,
  print_env = NA,
  print_trials = NA,
  help_model = NA,
  help_op = NA,
  test_output_scalars = NULL,
  test_sourcecode = NA,
  test_flags = NA
)

Arguments

Details

If MODEL is specified, Guild uses it instead of the default model.

OPERATION may alternatively be a Python script. In this case any current project is ignored and the script is run directly. Options in the format --NAME=VAL can be passed to the script using flags (see below).

⁠[MODEL]:OPERATION⁠ may be omitted if restart or proto is specified, in which case the operation used in RUN is used.

Specify FLAG values in the form FLAG=VAL.

Batch Files

One or more batch files can be used to run multiple trials by specifying the file path as ⁠@PATH⁠.

For example, to run trials specified in a CSV file named trials.csv, run:

guild run [MODEL:]OPERATION @trials.csv

NOTE: At this time you must specify the operation with batch files

• batch files only contain flag values and cannot be used to run different operations for the same command.

Batch files may be formatted as CSV, JSON, or YAML. Format is determined by the file extension.

Each entry in the file is used as a set of flags for a trial run.

CSV files must have a header row containing the flag names. Each subsequent row is a corresponding list of flag values that Guild uses for a generated trial.

JSON and YAML files must contain a top-level list of flag-to-value maps.

Use print_trials to preview the trials run for the specified batch files.

Flag Lists

A list of flag values may be specified using the syntax ⁠[VAL1[,VAL2]...]⁠. Lists containing white space must be quoted. When a list of values is provided, Guild generates a trial run for each value. When multiple flags have list values, Guild generates the cartesian product of all possible flag combinations.

Flag lists may be used to perform grid search operations.

For example, the following generates four runs for operation train and flags learning-rate and batch-size:

guild run train learning-rate[0.01,0.1] batch-size=[10,100]

You can preview the trials generated from flag lists using print_trials. You can save the generated trials to a batch file using save_trials. For more information, see PREVIEWING AND SAVING TRIALS below.

When optimizer is specified, flag lists may take on different meaning depending on the type of optimizer. For example, the random optimizer randomly selects values from a flag list, rather than generate trials for each value. See OPTIMIZERS for more information.

Optimizers

A run may be optimized using optimizer. An optimizer runs up to max_trials runs using flag values and flag configuration.

For details on available optimizers and their behavior, refer to https://guild.ai/optimizers/.

Limit Trials

When using flag lists or optimizers, which generate trials, you can limit the number of trials with max_trials. By default, Guild limits the number of generated trials to 20.

Guild limits trials by randomly sampling the maximum number from the total list of generated files. You can specify the seed used for the random sample with random_seed. The random seed is guaranteed to generate consistent results when used on the same version of Python. When used across different versions of Python, the results may be inconsistent.

Preview or Save Trials

When flag lists (used for grid search) or an optimizer is used, you can preview the generated trials using print_trials. You can save the generated trials as a CSV batch file using save_trials.

Start an Operation Using a Prototype Run

If proto is specified, Guild applies the operation, flags, and source code used in RUN to the new operation. You may add or redefine flags in the new operation. You may use an alternative operation, in which case only the flag values and source code from RUN are applied. RUN must be a run ID or unique run ID prefix.

Restart an Operation

If restart is specified, RUN is restarted using its operation and flags. Unlike proto, restart does not create a new run. You cannot change the operation, flags, source code, or run directory when restarting a run.

Staging an Operation

Use stage to stage an operation to be run later. Use start with the staged run ID to start it.

If start is specified, RUN is started using the same rules applied to restart (see above).

Alternate Run Directory

To run an operation outside of Guild's run management facility, use run_dir or stage-dir to specify an alternative run directory. These options are useful when developing or debugging an operation. Use stage-dir to prepare a run directory for an operation without running the operation itself. This is useful when you want to verify run directory layout or manually run an operation in a prepared directory.

NOTE: Runs started with run_dir are not visible to Guild and do not appear in run listings.

Control Visible GPUs

By default, operations have access to all available GPU devices. To limit the GPU devices available to a run, use gpus.

For example, to limit visible GPU devices to 0 and 1, run:

guild run gpus 0,1 ...

To disable all available GPUs, use no_gpus.

NOTE: gpus and no_gpus are used to construct the CUDA_VISIBLE_DEVICES environment variable used for the run process. If CUDA_VISIBLE_DEVICES is set, using either of these options redefines that environment variable for the run.

Optimize Runs

Use optimizer to run the operation multiple times in attempt to optimize a result. Use minimize or maximize to indicate what should be optimized. Use --max-runs to indicate the maximum number of runs the optimizer should generate.

Edit Flags

Use edit_flags to use an editor to review and modify flag values. Guild uses the editor defined in VISUAL or EDITOR environment variables. If neither environment variable is defined, Guild uses an editor suitable for the current platform.

Debug Source Code

Use debug_sourcecode to specify the location of project source code for debugging. Guild uses this path instead of the location of the copied soure code for the run. For example, when debugging project files, use this option to ensure that modules are loaded from the project location rather than the run directory.

Breakpoints

Use break to set breakpoints for Python based operations. LOCATION may be specified as ⁠[FILENAME:]LINE⁠ or as MODULE.FUNCTION.

If FILENAME is not specified, the main module is assumed. Use the value 1 to break at the start of the main module (line 1).

Relative file names are resolved relative to the their location in the Python system path. You can omit the .py extension.

If a line number does not correspond to a valid breakpoint, Guild attempts to set a breakpoint on the next valid breakpoint line in the applicable module.

guild_select_cli Select a run and shows its ID.

Description

This command is generally used when specifying a run ID for another Guild command. For example, to restart the latest train run:

Usage

guild_select_cli(
  ...,
  all = NA,
  min = NULL,
  max = NULL,
  short_id = NA,
  attr = NULL,
  path = NA,
  dir = NA,
  filter = NULL,
  operation = NULL,
  label = NULL,
  unlabeled = NA,
  tag = NULL,
  comment = NULL,
  marked = NA,
  unmarked = NA,
  started = NULL,
  digest = NULL,
  running = NA,
  completed = NA,
  error = NA,
  terminated = NA,
  pending = NA,
  staged = NA,
  remote = NULL
)

Arguments

Details

`guild run --restart $(guild select -o train)`

Specify a Run

You may specify a run using a run ID, a run ID prefix, or a one-based index corresponding to a run returned by the list command.

If RUN isn't specified, the latest matching run is selected.

Selecting Min or Max Scalar

To select the run with the lowest or highest column value, use min or max respectively. For example, to select the run with the lowest loss scalar value, use ⁠min loss⁠.

For help with COLSPEC formatting, see ⁠COLUMN SPECS⁠ in compare help by running ⁠guild compare help⁠.

Other run filters are applied before selecting a minimum or maximium scalar value.

Filter by Operation

Runs may be filtered by operation using operation. A run is only included if any part of its full operation name, including the package and model name, matches the value.

Use operation multiple times to include more runs.

Filter by Label

Use label to only include runs with labels containing a specified value. To select runs that do not contain a label, specify a dash '-' for VAL.

Use label multiple times to include more runs.

Filter by Tag

Use tag to only include runs with a specified tag. Tags must match completely and are case sensitive.

Use tag multiple times to include more runs.

Filter by Marked and Unmarked

Use marked to only include marked runs.

Use unmarked to only include unmarked runs. This option may not be used with marked.

Filter by Expression

Use filter to limit runs that match a filter expressions. Filter expressions compare run attributes, flag values, or scalars to target values. They may include multiple expressions with logical operators.

For example, to match runs with flag batch-size equal to 100 that have loss less than 0.8, use:

filter 'batch-size = 10 and loss < 0.8'

IMPORTANT: You must quote EXPR if it contains spaces or characters that the shell uses (e.g. '<' or '>').

Target values may be numbers, strings or lists containing numbers and strings. Strings that contain spaces must be quoted, otherwise a target string values does not require quotes. Lists are defined using square braces where each item is separated by a comma.

Comparisons may use the following operators: '=', '!=' (or '<>'), '<', '<=', '>', '>='. Text comparisons may use 'contains' to test for case-insensitive string membership. A value may be tested for membership or not in a list using 'in' or 'not in' respectively. An value may be tested for undefined using 'is undefined' or defined using 'is not undefined'.

Logical operators include 'or' and 'and'. An expression may be negated by preceding it with 'not'. Parentheses may be used to control the order of precedence when expressions are evaluated.

If a value reference matches more than one type of run information (e.g. a flag is named 'label', which is also a run attribute), the value is read in order of run attribute, then flag value, then scalar. To disambiguate the reference, use a prefix ⁠attr:⁠, ⁠flag:⁠, or ⁠scalar:⁠ as needed. For example, to filter using a flag value named 'label', use 'flag:label'.

Other examples:

⁠operation = train and acc > 0.9⁠ ⁠operation = train and (acc > 0.9 or loss < 0.3)⁠ ⁠batch-size = 100 or batch-size = 200⁠ ⁠batch-size in [100,200]⁠ ⁠batch-size not in [400,800]⁠ ⁠batch-size is undefined⁠ ⁠batch-size is not undefined⁠ ⁠label contains best and operation not in [test,deploy]⁠ ⁠status in [error,terminated]⁠

NOTE: Comments and tags are not supported in filter expressions at this time. Use comment and tag options along with filter expressions to further refine a selection.

Filter by Run Start Time

Use started to limit runs to those that have started within a specified time range.

IMPORTANT: You must quote RANGE values that contain spaces. For example, to filter runs started within the last hour, use the option:

started 'last hour'

You can specify a time range using several different forms:

⁠after DATETIME⁠ ⁠before DATETIME⁠ ⁠between DATETIME and DATETIME⁠ ⁠last N minutes|hours|days⁠ today|yesterday ⁠this week|month|year⁠ ⁠last week|month|year⁠ ⁠N days|weeks|months|years ago⁠

DATETIME may be specified as a date in the format YY-MM-DD (the leading ⁠YY-⁠ may be omitted) or as a time in the format HH:MM (24 hour clock). A date and time may be specified together as ⁠DATE TIME⁠.

When using ⁠between DATETIME and DATETIME⁠, values for DATETIME may be specified in either order.

When specifying values like minutes and hours the trailing s may be omitted to improve readability. You may also use min instead of minutes and hr instead of hours.

Examples:

⁠after 7-1⁠ ⁠after 9:00⁠ ⁠between 1-1 and 4-30⁠ ⁠between 10:00 and 15:00⁠ ⁠last 30 min⁠ ⁠last 6 hours⁠ today ⁠this week⁠ ⁠last month⁠ ⁠3 weeks ago⁠

Filter by Source Code Digest

To show runs for a specific source code digest, use -g or digest with a complete or partial digest value.

Filter by Run Status

Runs may also be filtered by specifying one or more status filters: running, completed, error, and terminated. These may be used together to include runs that match any of the filters. For example to only include runs that were either terminated or exited with an error, use ⁠terminated error⁠, or the short form -Set.

You may combine more than one status character with -S to expand the filter. For example, -Set shows only runs with terminated or error status.

Status filters are applied before RUN indexes are resolved. For example, a run index of 1 is the latest run that matches the status filters.

Select remote rnus

To apply select to remote runs, specify ⁠remote REMOTE⁠. Use ⁠guild remotes⁠ to list available remotes.

For information on configuring remotes, see ⁠guild remotes help⁠.

Launch Guild Viewer

Description

Launch Guild Viewer

Usage

guild_view(
  runs = NULL,
  ...,
  host = NULL,
  port = NULL,
  include_batch = FALSE,
  no_open = FALSE,
  stop = FALSE
)

Arguments

Value

The url where the Guild View application can be accessed.

The url where the View application can be accessed, invisibly.

Examples

## Not run: 
  guild_view()

  # see all supported options
  guild_view("--help")

  # three valid ways of supplying args to the guild executable
  guild_view("--port" = "5678")
  guild_view("--port", "5678")
  guild_view(c("--port", "5678"))

## End(Not run)

guild_view_cli Visualize runs in a local web application.

Description

Features include:

Usage

guild_view_cli(
  ...,
  host = NULL,
  port = NULL,
  include_batch = NA,
  no_open = NA,
  logging = NA,
  filter = NULL,
  operation = NULL,
  label = NULL,
  unlabeled = NA,
  tag = NULL,
  comment = NULL,
  marked = NA,
  unmarked = NA,
  started = NULL,
  digest = NULL,
  running = NA,
  completed = NA,
  error = NA,
  terminated = NA,
  pending = NA,
  staged = NA
)

Arguments

Details

• View and filter runs

• Compare runs

• Browse run files

• View run images and other media

• View run output

Guild View does not currently support starting or modifying runs. For these operations, use the applicable command line interface. Run 'guild help' for a complete list of commands.

By default Guild View shows all runs. You can filter runs using the command options described below.

Specify Runs

You may use one or more RUN arguments to indicate which runs apply to the command. RUN may be a run ID, a run ID prefix, or a one-based index corresponding to a run returned by the list command.

Indexes may also be specified in ranges in the form START:END where START is the start index and END is the end index. Either START or END may be omitted. If START is omitted, all runs up to END are selected. If END id omitted, all runs from START on are selected. If both START and END are omitted (i.e. the : char is used by itself) all runs are selected.

Filter by Operation

Runs may be filtered by operation using operation. A run is only included if any part of its full operation name, including the package and model name, matches the value.

Use operation multiple times to include more runs.

Filter by Label

Use label to only include runs with labels containing a specified value. To select runs that do not contain a label, specify a dash '-' for VAL.

Use label multiple times to include more runs.

Filter by Tag

Use tag to only include runs with a specified tag. Tags must match completely and are case sensitive.

Use tag multiple times to include more runs.

Filter by Marked and Unmarked

Use marked to only include marked runs.

Use unmarked to only include unmarked runs. This option may not be used with marked.

Filter by Expression

Use filter to limit runs that match a filter expressions. Filter expressions compare run attributes, flag values, or scalars to target values. They may include multiple expressions with logical operators.

For example, to match runs with flag batch-size equal to 100 that have loss less than 0.8, use:

filter 'batch-size = 10 and loss < 0.8'

IMPORTANT: You must quote EXPR if it contains spaces or characters that the shell uses (e.g. '<' or '>').

Target values may be numbers, strings or lists containing numbers and strings. Strings that contain spaces must be quoted, otherwise a target string values does not require quotes. Lists are defined using square braces where each item is separated by a comma.

Comparisons may use the following operators: '=', '!=' (or '<>'), '<', '<=', '>', '>='. Text comparisons may use 'contains' to test for case-insensitive string membership. A value may be tested for membership or not in a list using 'in' or 'not in' respectively. An value may be tested for undefined using 'is undefined' or defined using 'is not undefined'.

Logical operators include 'or' and 'and'. An expression may be negated by preceding it with 'not'. Parentheses may be used to control the order of precedence when expressions are evaluated.

If a value reference matches more than one type of run information (e.g. a flag is named 'label', which is also a run attribute), the value is read in order of run attribute, then flag value, then scalar. To disambiguate the reference, use a prefix ⁠attr:⁠, ⁠flag:⁠, or ⁠scalar:⁠ as needed. For example, to filter using a flag value named 'label', use 'flag:label'.

Other examples:

⁠operation = train and acc > 0.9⁠ ⁠operation = train and (acc > 0.9 or loss < 0.3)⁠ ⁠batch-size = 100 or batch-size = 200⁠ ⁠batch-size in [100,200]⁠ ⁠batch-size not in [400,800]⁠ ⁠batch-size is undefined⁠ ⁠batch-size is not undefined⁠ ⁠label contains best and operation not in [test,deploy]⁠ ⁠status in [error,terminated]⁠

NOTE: Comments and tags are not supported in filter expressions at this time. Use comment and tag options along with filter expressions to further refine a selection.

Filter by Run Start Time

Use started to limit runs to those that have started within a specified time range.

IMPORTANT: You must quote RANGE values that contain spaces. For example, to filter runs started within the last hour, use the option:

started 'last hour'

You can specify a time range using several different forms:

⁠after DATETIME⁠ ⁠before DATETIME⁠ ⁠between DATETIME and DATETIME⁠ ⁠last N minutes|hours|days⁠ today|yesterday ⁠this week|month|year⁠ ⁠last week|month|year⁠ ⁠N days|weeks|months|years ago⁠

DATETIME may be specified as a date in the format YY-MM-DD (the leading ⁠YY-⁠ may be omitted) or as a time in the format HH:MM (24 hour clock). A date and time may be specified together as ⁠DATE TIME⁠.

When using ⁠between DATETIME and DATETIME⁠, values for DATETIME may be specified in either order.

When specifying values like minutes and hours the trailing s may be omitted to improve readability. You may also use min instead of minutes and hr instead of hours.

Examples:

⁠after 7-1⁠ ⁠after 9:00⁠ ⁠between 1-1 and 4-30⁠ ⁠between 10:00 and 15:00⁠ ⁠last 30 min⁠ ⁠last 6 hours⁠ today ⁠this week⁠ ⁠last month⁠ ⁠3 weeks ago⁠

Filter by Source Code Digest

To show runs for a specific source code digest, use -g or digest with a complete or partial digest value.

Filter by Run Status

Runs may also be filtered by specifying one or more status filters: running, completed, error, and terminated. These may be used together to include runs that match any of the filters. For example to only include runs that were either terminated or exited with an error, use ⁠terminated error⁠, or the short form -Set.

You may combine more than one status character with -S to expand the filter. For example, -Set shows only runs with terminated or error status.

Status filters are applied before RUN indexes are resolved. For example, a run index of 1 is the latest run that matches the status filters.

Install guildai core

Description

This installs the guild executable for use by the R package. It creates an isolated python virtual environment private to the R package and installs guildai into it. Repeated calls to install_guild() result in a fresh installation.

Usage

install_guild(guildai = "guildai", python = find_python())

Arguments

Details

It requires that a suitable python version is available on the system.

Value

path to the guild executable

Note

install_guild() installs guild as an isolated VM. For guild to run a python operation, the python package guildai must be installed in the python library where it will be used, E.g., with ⁠pip install guildai⁠ or reticulate::py_install().

Examples

## Not run: 
## Install release version:
install_guild()

## Install release version using a specific python
# path_to_python <- reticulate::install_python() # path to python executable
install_guild("guildai", python = path_to_python)

## Install development version
install_guild(guildai = "dev", python = path_to_python)

## Install development version from URL
 install_guild(
   guildai = "https://api.github.com/repos/guildai/guildai/tarball/HEAD",
   python = path_to_python)

## Install local development version:
install_guild(c("-e", "~/guild/guildai"))

## End(Not run)

Install guild for usage in the Terminal

Description

This function makes available the guild executable installed by install_guild() for usage in the Terminal.

Usage

install_guild_cli(
  dest = "~/bin",
  completions = basename(Sys.getenv("SHELL")) %in% c("bash", "zsh", "fish")
)

Arguments

Details

Note that the guild executable installed by the R function install_guild() is not able to run python operations. To run python operations with guild, you must install guild into the target python installation with ⁠pip install guildai⁠, and ensure that the desired guild executable is on the PATH.

Value

path to the installed guild executable, invisibly.

Is code executing in the context of a guild run?

Description

Is code executing in the context of a guild run?

Usage

is_run_active()

Value

Boolean

Resolve run ids

Description

This is a equivalent to runs_info(...)$id, implemented more efficiently.

Usage

resolve_run_ids(runs = NULL, ..., all = TRUE)

Arguments

Details

guild supports a rich syntax for runs selection throughout the api. The same selection syntax is shared by the ⁠runs_*⁠ family of functions: runs_info(), runs_scalars(), runs_comment(), runs_label(), runs_mark(), runs_tag() runs_delete(),runs_purge(), runs_restore(), runs_export(), runs_import().

Value

A character vector of run ids.

Note

You can call Sys.setenv(GUILD_DEBUG_R = 1) to see what system calls to the guild executable are made. This is useful when looking to understand how R arguments are transformed into a cli system call.

Examples

## Not run: 
resolve_run_ids() # returns all run ids.
resolve_run_ids(1) # last run
resolve_run_ids(1:2) # last 2 runs
resolve_run_ids(1:2, operation = "train.py")

# three ways of getting ids for the currently staged or running runs
resolve_run_ids(staged = TRUE, running = TRUE)
resolve_run_ids("--staged", "--running")
resolve_run_ids(c("--staged", "--running"))
resolve_run_ids(I("--staged --running"))

# resolve_run_ids() uses the same selection rules and syntax as runs_info()
stopifnot(identical(
  resolve_run_ids(),
  runs_info()$id
))

## End(Not run)

Delete runs

Description

Delete runs

Usage

runs_delete(runs = NULL, ...)

runs_purge(runs = NULL, ...)

runs_restore(runs = NULL, ...)

Arguments

Details

• runs_delete() moves runs into a guild managed "trash" directory.

• runs_restore() moves runs back into the main guild managed "runs" directory.

• runs_purge() permanently delete runs from "trash" directory. Only deleted runs can be purged.

Value

The value supplied to the runs argument, invisibly.

Note

To see deleted runs, do guildai:::guild("runs list --deleted") (runs_info("--deleted") supported soon)

Move or copy runs

Description

Move or copy runs

Usage

runs_export(runs = NULL, location, ..., move = FALSE, copy_resources = FALSE)

runs_import(runs = NULL, location, ..., move = FALSE, copy_resources = FALSE)

Arguments

Value

The value supplied to the runs argument, invisibly.

Get runs information

Description

Returns a dataframe with information about the guild runs stored in guild home. Guild home is determined either by consulting the env var Sys.getenv("GUILD_HOME"), or if unset, by looking for a .guild directory, starting from the current working directory and walking up parent directories up to ~ or /.

Usage

runs_info(
  runs = NULL,
  ...,
  filter = NULL,
  operation = NULL,
  label = NULL,
  unlabeled = NA,
  tag = NULL,
  comment = NULL,
  marked = NA,
  unmarked = NA,
  started = NULL,
  digest = NULL,
  running = NA,
  completed = NA,
  error = NA,
  terminated = NA,
  pending = NA,
  staged = NA,
  deleted = NA,
  include_batch = NA
)

Arguments

Details

Guild has support for a custom filter expression syntax. This syntax is primarily useful in the terminal, and R users will generally prefer to filter the returned dataframe directly using dplyr::filter() or [. Nevertheless, R users can supply guild filter expressions here as well.

Filter by Expression

Use filter to limit runs that match a filter expressions. Filter expressions compare run attributes, flag values, or scalars to target values. They may include multiple expressions with logical operators.

For example, to match runs with flag batch-size equal to 100 that have loss less than 0.8, use:

runs_info(filter = "batch-size = 10 and loss < 0.8")

Target values may be numbers, strings or lists containing numbers and strings. Lists are defined using square braces where each item is separated by a comma.

Comparisons may use the following operators: '=', '!=', '<', '<=', '>', '>='.

Text comparisons may use 'contains' to test for case-insensitive string membership. A value may be tested for membership or not in a list using 'in' or 'not in' respectively. An value may be tested for undefined using 'is undefined' or defined using 'is not undefined'.

Logical operators include 'or' and 'and'. An expression may be negated by preceding it with 'not'. Parentheses may be used to control the order of precedence when expressions are evaluated.

If a value reference matches more than one type of run information (e.g. a flag is named 'label', which is also a run attribute), the value is read in order of run attribute, then flag value, then scalar. To disambiguate the reference, use a prefix ⁠attr:⁠, ⁠flag:⁠, or ⁠scalar:⁠ as needed. For example, to filter using a flag value named 'label', use 'flag:label'.

Other examples:

"operation = train and acc > 0.9"
"operation = train and (acc > 0.9 or loss < 0.3)"
"batch-size = 100 or batch-size = 200"
"batch-size in [100,200]"
"batch-size not in [400,800]"
"batch-size is undefined"
"batch-size is not undefined"
"label contains best and operation not in [test,deploy]"
"status in [error,terminated]"

NOTE: Comments and tags are not supported in filter expressions at this time. Use comment and tag options along with filter expressions to further refine a selection.

Filter by Run Start Time

Use started to limit runs to those that have started within a specified time range.

runs_info(started = 'last hour')

You can specify a time range using several different forms:

"after DATETIME"
"before DATETIME"
"between DATETIME and DATETIME"
"last N minutes|hours|days"
"today|yesterday"
"this week|month|year"
"last week|month|year"
"N days|weeks|months|years ago"

DATETIME may be specified as a date in the format YY-MM-DD (the leading ⁠YY-⁠ may be omitted) or as a time in the format HH:MM (24 hour clock). A date and time may be specified together as ⁠DATE TIME⁠.

When using ⁠between DATETIME and DATETIME⁠, values for DATETIME may be specified in either order.

When specifying values like minutes and hours the trailing s may be omitted to improve readability. You may also use min instead of minutes and hr instead of hours.

Examples:

"after 7-1"
"after 9:00"
"between 1-1 and 4-30"
"between 10:00 and 15:00"
"last 30 min"
"last 6 hours"
"today"
"this week"
"last month"
"3 weeks ago"

Filter by Run Status

Runs may also be filtered by specifying one or more status filters: running, completed, error, and terminated. These may be used together to include runs that match any of the filters. For example to only include runs that were either terminated or exited with an error, use

runs_info(terminated = TRUE, error = TRUE)

Status filters are applied before RUN indexes are resolved. For example, a run index of 1 (as in, runs_info(1, terminated = TRUE, error = TRUE) is the latest run that matches the status filters.

Value

A dataframe (tibble) of runs

Examples

## Not run: 
withr::with_package("dplyr", {

runs_info() # get the full set of runs
runs_info(1) # get the most recent run
runs_info(1:3) # get the last 3 runs

# some other examples for passing filter expressions
runs_info(staged = TRUE) # list only staged runs
runs_info(tag = c("convnet", "keras"), started = "last hour")
runs_info(error = TRUE)

runs <- runs_info()

# filter down the runs list to ones of interest
runs <- runs %>%
  filter(exit_status == 0) %>% # run ended without an error code
  filter(scalars$test_accuracy > .8) %>%
  filter(flags$epochs > 10) %>%
  arrange(scalars$test_loss) %>%
  select(id, flags, scalars)

# retrieve full scalars history from the runs of interest
runs$id %>%
  runs_scalars()

# export the best run
best_runs_dir <- tempfile()
dir.create(best_runs_dir)
runs %>%
  slice_max(scalars$test_accuracy) %>%
  runs_tag("best") %>%
  runs_export(best_runs_dir)

})

## End(Not run)

Annotate runs

Description

Annotate runs

Usage

runs_label(runs = NULL, label = NULL, ..., clear = FALSE)

runs_tag(runs = NULL, add = NULL, ..., remove = NULL, clear = FALSE)

runs_mark(runs = NULL, ..., clear = FALSE)

runs_comment(runs = NULL, comment = NULL, ..., delete = NULL, clear = FALSE)

Arguments

Details

Annotation types and their recommended uses:

• labels: short, single line descriptions tailored for readability, not programmatic consumption. Labels are presented prominently in guild_view() and other run views.

• tags: short single-token strings. Tags can be used for organizing, grouping, and filtering runs.

• comments: longer (potentially multi-paragraph) descriptions of the run. Guild stores and presents run comments as log entries, complete with timestamps and author info.

• marks: A boolean attribute of a run (a run can be marked or unmarked). Marked runs are primarily used to declare a run as the preferred source for resolving an operation dependency. If a operation declares a dependency on another operation, and one of the dependent operation runs is marked, the marked run is used rather than the latest run for resolving the dependency. Marks can also be a convenient mechanism for ad-hoc filtering operations, but in general, tags are preferred over marks for this.

Value

The value supplied to the runs argument, invisibly.

Note

runs_comment() will open up an editor if comment is not supplied.

Examples

## Not run: 
runs_info(1) %>% runs_tag(clear = TRUE)
runs_info(1) %>% runs_tag("foo")
runs_info(1)$tags
runs_info(1) %>% runs_tag("bar")
runs_info(1)$tags
runs_info(1) %>% runs_tag(remove = "foo")
runs_info(1)$tags
runs_info(1) %>% runs_tag("baz", clear = TRUE)
runs_info(1)$tags

## pass through options to `guild tag` cli subcommand
runs_tag("--help")

## End(Not run)

Get full set of runs scalars

Description

Get full set of runs scalars

Usage

runs_scalars(runs = NULL, ...)

Arguments

Value

A dataframe (tibble) of runs

Examples

## Not run: 
runs_scalars(1) # scalars from most recent run
runs_scalars(1:2) # scalars form two most recent runs

# pass in a dataframe of runs
runs_info() %>%
  filter(flags$epochs > 5) %>%
  runs_scalars()

## End(Not run)

Render a Run Summary Report

Description

Render a Run Summary Report

Usage

view_run_report(
  run = "1",
  output_dir = file.path(tempdir(), id),
  template = system.file("templates/view-run.qmd", package = "guildai"),
  viewer = getOption("guildai.viewer"),
  ...
)

Arguments

Value

path to the generated html, invisibly

View a runs diff

Description

Renders and displays an HTML document showing the diffs between two run directories.

Usage

view_runs_diff(
  runs = 1:2,
  ...,
  output_html = tempfile("guild-diff-", fileext = ".html"),
  exclude = ".guild",
  output_empty = TRUE,
  exclude_directory_metadata = c("yes", "no", "auto", "recursive"),
  viewer = utils::browseURL
)

Arguments

Value

path to the generated html, invisibly

Write run attributes

Description

This function does nothing if is_run_active() is FALSE.

Usage

write_run_attr(name, data, ..., append = FALSE)

Arguments

Value

the written lines, invisibly.