Keepsake Version control for machine learning

This is the reference for the Keepsake command-line interface. You can also see this in the terminal by running keepsake --help or keepsake command --help.

To install the command-line interface, see the installation instructions.

Commands

keepsake analytics – Enable or disable analytics
keepsake checkout – Copy files from an experiment or checkpoint into the project directory
keepsake diff – Compare two experiments or checkpoints
keepsake feedback – Submit feedback to the team!
keepsake ls – List experiments in this project
keepsake ps – List running experiments in this project
keepsake rm – Remove experiments or checkpoint
keepsake show – View information about an experiment or checkpoint

`keepsake analytics`

The Keepsake CLI sends anonymous analytics about commands you run.

The following data is sent on each command line invokation:

A random token for the machine (e.g. "9f3027bb-0eb8-917d-e5bf-c6c1bdb1fd0a")
The subcommand you ran, without any options or arguments (e.g. "keepsake run", not "keepsake run python secretproject.py")
The Keepsake version (e.g. "1.0.0")
Your CPU architecture (e.g. "amd64")
Your operating system (e.g. "linux")

To learn more, please refer to https://keepsake.ai/docs/learn/analytics

These analytics really help us, and we'd really appreciate it if you left it on. But, if you want to opt out, you can run this command:

keepsake analytics off

Usage

keepsake analytics <on|off> [flags]

Flags

-h, --help   help for analytics
      --color                      Display color in output (default true)
  -D, --project-directory string   Project directory. Default: nearest parent directory with keepsake.yaml
  -v, --verbose                    Verbose output

`keepsake checkout`

Copy files from an experiment or checkpoint into the project directory

Usage

keepsake checkout <experiment or checkpoint ID> [flags]

Flags

-f, --force                     Force checkout without prompt, even if the directory is not empty
  -h, --help                      help for checkout
  -o, --output-directory string   Output directory (defaults to working directory or directory with keepsake.yaml in it)
      --path string               A specific file or directory to checkout (defaults to all files or directory in checkpoint/experiment)
  -R, --repository string         Repository URL (e.g. 's3://my-keepsake-bucket' (if omitted, uses repository URL from keepsake.yaml)
      --color                      Display color in output (default true)
  -D, --project-directory string   Project directory. Default: nearest parent directory with keepsake.yaml
  -v, --verbose                    Verbose output

`keepsake diff`

Compare two experiments or checkpoints.

If an experiment ID is passed, it will pick the best checkpoint from that experiment. If a primary metric is not defined in keepsake.yaml, it will use the latest checkpoint.

Usage

keepsake diff <ID> <ID> [flags]

Flags

-h, --help                help for diff
  -R, --repository string   Repository URL (e.g. 's3://my-keepsake-bucket' (if omitted, uses repository URL from keepsake.yaml)
      --color                      Display color in output (default true)
  -D, --project-directory string   Project directory. Default: nearest parent directory with keepsake.yaml
  -v, --verbose                    Verbose output

`keepsake feedback`

Submit feedback to the team!

Usage

keepsake feedback [flags]

Flags

-h, --help   help for feedback
      --color                      Display color in output (default true)
  -D, --project-directory string   Project directory. Default: nearest parent directory with keepsake.yaml
  -v, --verbose                    Verbose output

`keepsake ls`

List experiments in this project

Usage

keepsake ls [flags]

Examples

List all experiments in the current project:
$ keepsake ls
List experiments that have run for 50 steps or less:
$ keepsake ls --filter "step <= 50"
List experiments where the parameter "optimizer" is "adam" and
the best "accuracy" metric is greater than 0.8:
$ keepsake ls --filter "optimizer = adam" --filter "accuracy > 0.8"
Sort all stopped experiments by the metric "val_loss":
$ keepsake ls --sort "val_loss" --filter "status = stopped"

Flags

--all                  Output all params and metrics. Default: only params/metrics that differ
  -f, --filter stringArray   Filters (format: "<name> <operator> <value>")
  -h, --help                 help for ls
      --json                 Print output in JSON format
  -q, --quiet                Only print experiment IDs
  -R, --repository string    Repository URL (e.g. 's3://my-keepsake-bucket' (if omitted, uses repository URL from keepsake.yaml)
  -s, --sort string          Sort key. Suffix with '-desc' for descending sort, e.g. --sort=created-desc (default "created")
      --color                      Display color in output (default true)
  -D, --project-directory string   Project directory. Default: nearest parent directory with keepsake.yaml
  -v, --verbose                    Verbose output

`keepsake ps`

List running experiments in this project

Usage

keepsake ps [flags]

Flags

--all                  Output all params and metrics. Default: only params/metrics that differ
  -f, --filter stringArray   Filters (format: "<name> <operator> <value>")
  -h, --help                 help for ps
      --json                 Print output in JSON format
  -q, --quiet                Only print experiment IDs
  -R, --repository string    Repository URL (e.g. 's3://my-keepsake-bucket' (if omitted, uses repository URL from keepsake.yaml)
  -s, --sort string          Sort key. Suffix with '-desc' for descending sort, e.g. --sort=created-desc (default "created")
      --color                      Display color in output (default true)
  -D, --project-directory string   Project directory. Default: nearest parent directory with keepsake.yaml
  -v, --verbose                    Verbose output

`keepsake rm`

Remove experiments or checkpoints.

To remove experiments or checkpoints, pass any number of IDs (or prefixes).

Usage

keepsake rm <experiment or checkpoint ID> [experiment or checkpoint ID...] [flags]

Examples

Delete an experiment and its checkpoints
(where a1b2c3d4 is an experiment ID):
keepsake rm a1b2c3d4
Delete all experiments where the metric "val_accuracy" is less
than 0.2 at the best checkpoints:
keepsake rm $(keepsake ls -q --filter "val_accuracy < 0.2")

Flags

-f, --force               Force delete without interactive prompt
  -h, --help                help for rm
  -R, --repository string   Repository URL (e.g. 's3://my-keepsake-bucket' (if omitted, uses repository URL from keepsake.yaml)
      --color                      Display color in output (default true)
  -D, --project-directory string   Project directory. Default: nearest parent directory with keepsake.yaml
  -v, --verbose                    Verbose output

`keepsake show`

View information about an experiment or checkpoint

Usage

keepsake show <experiment or checkpoint ID> [flags]

Flags

--all                 Show all information
  -h, --help                help for show
      --json                Print output in JSON format
  -R, --repository string   Repository URL (e.g. 's3://my-keepsake-bucket' (if omitted, uses repository URL from keepsake.yaml)
      --color                      Display color in output (default true)
  -D, --project-directory string   Project directory. Default: nearest parent directory with keepsake.yaml
  -v, --verbose                    Verbose output