Compute¤

substrax.compute runs a project's jobs on a compute backend: this machine, Modal, or any cloud SkyPilot reaches. A job is declared once, in the project's pyproject.toml, and runs the same way on every backend, because every backend runs the same program on the remote machine: the worker.

Declaring jobs¤

[tool.substrax.compute]
backend = "modal"                        # used when the command line names none

[tool.substrax.compute.backends.modal]   # passed to the backend as its settings
outputs_volume = "demo-outputs"

[tool.substrax.compute.jobs.examples]
examples = ["examples/metrics"]          # one task per script
commands = { gpu-tests = ["python", "-m", "pytest", "tests/gpu"] }
extras = ["cuda12"]
resources = { accelerator = { kind = "L4", count = 1 }, cpu = 8, memory_mib = 32768 }
timeout_seconds = 1800
task_timeout_seconds = 300
runtime = { platforms = ["cuda"], xla_flags = ["--xla_gpu_deterministic_ops=true"] }
env = { TF_CPP_MIN_LOG_LEVEL = "1" }
mounts = [{ name = "datasets", path = "/data", access = "read-only" }]

A misspelled key is refused, not ignored. resources is what the machine must have: an accelerator, CPU cores (cpu, fractional allowed) and memory in MiB (memory_mib), each the least it gets. Modal takes them as gpu, cpu and memory; SkyPilot as accelerators, cpus = "<n>+" and memory = "<n>MB+"; the local backend provisions nothing. --accelerator replaces the configured accelerator for one run. examples lists each directory's scripts as substrax.examples.discover_examples does, skipping every name that starts with _; a task's leading "python" is the project environment's interpreter.

Running them¤

uv run substrax-compute run examples                  # submit, follow, wait, fetch
uv run substrax-compute run examples --accelerator H100:2 --detach
uv run substrax-compute status                        # every stored run
uv run substrax-compute logs <run-id> --follow
uv run substrax-compute fetch <run-id> --into ~/runs
uv run substrax-compute cancel <run-id>

The command runs from the project's own environment, whose lock must include substrax: the worker is the project's locked substrax. A run's handle is kept under the state directory ($SUBSTRAX_STATE_DIR, else $XDG_STATE_HOME/substrax), so a detached run is found again from any shell. Fetched outputs go to --into or under the state directory, never into the project.

What a run produces¤

<run-id>/
  manifest.json            # the devices JAX saw; every task: argv, status, exit code, seconds, files
  devices.log              # only when the device probe failed
  <task>/stdout.log
  <task>/stderr.log
  <task>/<name>/...        # what the task saved through resolve_output_dir(name)

Each task runs from the project root with AVITAI_OUTPUT_DIR set to its own directory and the job's JaxRuntime applied, for at most its own budget or what is left of the job's. A failed task does not stop the ones after it. The worker rewrites the manifest after every task, and marks it finished after the last. Before the first task a child with the job's JaxRuntime reports the platform and each device's kind, which the manifest records: what the provider allocated, which can differ from the request (an H100:8 request has run on H200). The worker itself never starts a JAX backend.

Backends¤

Backend	Runs on	Needs
`local`	this machine, in the project's current environment	nothing
`modal`	Modal, one detached function per run, outputs on a Volume	`substrax[modal]`, a Modal account, and a workspace image builder of 2025.06 or later
`skypilot`	a cluster SkyPilot provisions for the run on any cloud it supports; inputs and outputs on buckets, or copied	SkyPilot installed as its documentation says (`uv tool install --with pip "skypilot[gcp]"`, or the provider's extra), and the transport's tools: `gcloud` or `aws` for buckets, `ssh` and `rsync` for copies

The Modal image is built in layers keyed on what they depend on: the locked dependencies (Image.uv_sync), then the project's source, then the project installed without dependencies, so an edit to the source rebuilds only the last two. Image.uv_sync needs image builder 2025.06 or later, set for the workspace in Modal's dashboard under Settings → Image config; an older builder fails the build with Image builder version <= 2024.10 requires modal to be specified in your pyproject.toml file. The image runs the Python this interpreter runs, which the serialized worker requires, and which uv sync is told to use so a project's own python-preference cannot choose another.

SkyPilot¤

SkyPilot is never imported into the project's environment. Every operation (launch, queue, logs, cancel, teardown) runs substrax/compute/sky_sdk.py in SkyPilot's own interpreter, which calls SkyPilot's public SDK (0.11.1) and reads its native request and queue records. Before a launch waits for provisioning, it writes the run's intent and SkyPilot's request id beside the task file; an interrupted launch is reconciled from them, not submitted again. cancel leaves a run that already ended as it is; otherwise it cancels the request and the job and tears the cluster down, attempting every step and reporting each failure (substrax-compute cancel exits 1).

The settings under [tool.substrax.compute.backends.skypilot], all optional:

Setting	Default	Meaning
`infra`	any enabled cloud	where to run, in SkyPilot's `infra` syntax: `gcp`, `gcp/us-central1`, `runpod`
`use_spot`	`false`	run on spot instances
`setup_commands`	`[]`	commands that prepare the image before the locked environment is synced
`sky`	`"sky"`	the `sky` command, used to find SkyPilot's interpreter beside it
`sky_python`	the `python` beside `sky`	SkyPilot's interpreter
`worker_python`	`"uv run --no-sync python"`	the command that starts Python in the synced environment
`state_dir`	the default state directory	where task files, launch records and cancellations are kept

A run's files move one of two ways, configured as exactly one of these tables.

Bucket transport ([...skypilot.bucket]): SkyPilot mounts the buckets, so the outputs outlive the cluster, which SkyPilot removes when the job ends. Providers that cannot mount buckets (RunPod, Vast) are refused.

[tool.substrax.compute.backends.skypilot.bucket]
outputs_bucket = "gs://avitai-runs"                # required; gs:// or s3://
buckets = { datasets = "gs://avitai-datasets" }    # each mount's bucket, by mount name

Copy transport ([...skypilot.copy]): SkyPilot copies each mount's local source onto the cluster, read-only; a writable mount is refused, since a copy cannot be written back. The worker runs under bubblewrap, so the image has to provide bwrap with working user namespaces: the system, the project and the inputs are read-only, /tmp and the outputs writable. fetch copies the outputs over rsync and checks them against manifests of the remote directory taken before and after; a difference keeps the cluster up. Fetch the outputs before SkyPilotBackend.teardown(handle), which removes the cluster, and with it the outputs on it. lifetime_minutes starts once setup ends; it is not a provisioning deadline or a cost cap.

[tool.substrax.compute.backends.skypilot.copy]
sources = { datasets = "/data/datasets" }   # each mount's local path, by mount name
disk_size = 60                              # GB
lifetime_minutes = 60                       # idle time after setup before the cluster removes itself
network_isolation = false                   # true runs the worker without network access

Run handles¤

A run's handle records the backend settings it was submitted with (RunHandle.settings), and status, logs, fetch and cancel rebuild the backend from them, never from the project's current configuration; a backend passes them to new_run(spec, project, settings=...). Handles are of version 2. One of another version is refused: substrax-compute status RUN_ID exits 2, and substrax-compute status lists every run it can read, names each record it cannot with the reason, and exits 1.

Adding a provider¤

A backend is any object that satisfies ComputeBackend and is built from its settings table. Register the factory in the substrax.compute.backends entry-point group:

[project.entry-points."substrax.compute.backends"]
myprovider = "mypackage.backend:MyBackend"

and inherit the contract every backend must pass:

from pathlib import Path

from substrax.compute.backend import ComputeBackend
from substrax.testing.compute import BackendContract

from mypackage.backend import MyBackend


class TestMyBackend(BackendContract):
    def make_backend(self, state_dir: Path) -> ComputeBackend:
        return MyBackend({"state_dir": str(state_dir)})

`substrax.compute` ¤

Run a project's commands on remote accelerators through provider plugins.

A JobSpec states what runs: tasks, the machine's resources (accelerators, CPU cores, memory), extras, JAX settings and mounts. The worker (python -m substrax.compute.worker) runs it the same way on every provider. Importing this package imports no jax and no provider library.

`JOB_SPEC_VERSION = 2` `module-attribute` ¤

The job spec layout this release writes and reads.

`Accelerator(*, kind, count=1)` `dataclass` ¤

The accelerators a job runs on.

Attributes:

Name	Type	Description
`kind`	`str`	The accelerator's name as the providers spell it, such as `"L4"`, `"A100-80GB"` or `"H100"`; a backend refuses a kind its provider does not offer.
`count`	`int`	How many.

`__post_init__()` ¤

Validate the kind and count.

Raises:

Type	Description
`ValueError`	If `kind` is not one word or `count` is below one.

`JobSpec(*, schema_version=JOB_SPEC_VERSION, name, tasks, timeout_seconds, resources=Resources(), extras=(), runtime=JaxRuntime(), env=dict(), mounts=())` `dataclass` ¤

A job: its tasks and everything the machine that runs them must provide.

Attributes:

Name	Type	Description
`schema_version`	`int`	The layout of this record; see :data:`JOB_SPEC_VERSION`.
`name`	`str`	The job's name, such as `"examples"`.
`tasks`	`tuple[Task, ...]`	The commands, run in order; a failed task does not stop the ones after it.
`timeout_seconds`	`float`	Seconds the whole job may take; the backend stops it after that.
`resources`	`Resources`	The accelerators, CPU cores and memory the machine must have.
`extras`	`tuple[str, ...]`	The project's optional dependency groups installed with its locked dependencies.
`runtime`	`JaxRuntime`	The JAX settings of every task.
`env`	`Mapping[str, str]`	Variables set in every task before `runtime` is applied.
`mounts`	`tuple[Mount, ...]`	The named storage the tasks read or write, besides the run's outputs.

`__post_init__()` ¤

Validate the job as a whole.

Raises:

Type	Description
`ValueError`	If the version is not this release's, the name is not one path component, there are no tasks, the timeout is not a positive finite number, or two tasks or two mounts share a name, or two mounts a path.

`Mount(*, name, path, access)` `dataclass` ¤

Named storage a job sees at a path; each backend maps the name to its own storage.

Attributes:

Name	Type	Description
`name`	`str`	The storage's name in the backend's settings, such as `"datasets"`.
`path`	`str`	The absolute directory the job sees it at.
`access`	`MountAccess`	Whether the job may write to it.

`__post_init__()` ¤

Validate the name and path.

Raises:

Type	Description
`ValueError`	If the name is not one path component, or the path is not an absolute, normalised directory below the root.

`MountAccess` ¤

Bases: StrEnum

Whether a job may change what a mount holds.

`Resources(*, accelerator=None, cpu=None, memory_mib=None)` `dataclass` ¤

What the machine a job runs on must have; each request is the least it gets.

Attributes:

Name	Type	Description
`accelerator`	`Accelerator \| None`	The accelerators; `None` runs on CPUs.
`cpu`	`float \| None`	The CPU cores, fractional allowed; `None` takes the provider's default.
`memory_mib`	`int \| None`	The memory, in MiB; `None` takes the provider's default.

`__post_init__()` ¤

Validate the requests.

Raises:

Type	Description
`ValueError`	If a CPU or memory request is not a positive finite number.

`Task(*, name, argv, timeout_seconds)` `dataclass` ¤

One command a job runs, from the project's root, in the project's environment.

Attributes:

Name	Type	Description
`name`	`str`	The task's name, and the directory its logs and outputs go to.
`argv`	`tuple[str, ...]`	The command. A leading `"python"` is the project environment's interpreter.
`timeout_seconds`	`float`	Seconds before the task is killed.

`__post_init__()` ¤

Validate the name, command and timeout.

Raises:

Type	Description
`ValueError`	If the name is not one path component, the command is empty, or the timeout is not a positive finite number.

`example_tasks(paths, *, project, timeout_seconds)` ¤

Make one task per example script, run by the project's interpreter from its root.

Parameters:

Name	Type	Description	Default
`paths`	`Iterable[Path]`	The examples, such as :func:`~substrax.examples.discover_examples` lists them.	required
`project`	`Path`	The project root the paths lie under.	required
`timeout_seconds`	`float`	Each example's budget.	required

Returns:

Type	Description
`Task`	The tasks, in the order of `paths`, each named by its path relative to `project` with
`...`	dots for separators and without the suffix.

Raises:

Type	Description
`ValueError`	If a path lies outside `project`.

`read_job_spec(data)` ¤

Read a job spec from its parsed JSON object.

Parameters:

Name	Type	Description	Default
`data`	`Mapping[str, JsonValue]`	The JSON object `dump_record` wrote.	required

Returns:

Type	Description
`JobSpec`	The spec.

Raises:

Type	Description
`ValueError`	If the spec's version is not this release's (the message names both), or a field is missing or holds a value its annotation does not admit.

`substrax.compute.backend` ¤

The interface every compute provider implements, and the handles that name submitted runs.

A backend turns a :class:~substrax.compute.JobSpec into a run on its provider and reports on it afterwards. It returns a :class:RunHandle from :meth:ComputeBackend.submit and needs nothing else to find the run again, so a run submitted from one shell can be followed, fetched or cancelled from another. :class:RunStore keeps the handles.

`SPEC_NAME = 'spec.json'` `module-attribute` ¤

The file a staged run's spec is written to.

`STATE_DIR_ENV = 'SUBSTRAX_STATE_DIR'` `module-attribute` ¤

Overrides the directory substrax keeps run handles and local runs in.

`BackendFactory = Callable[[Mapping[str, JsonValue]], ComputeBackend]` `module-attribute` ¤

What a backend's entry point loads: it builds the backend from its settings table.

`RunState` ¤

Bases: StrEnum

Where a run is.

`is_final` `property` ¤

Whether the run has ended.

`RunHandle(*, schema_version=HANDLE_VERSION, run_id, backend, job, project, submitted_at, settings, details=dict())` `dataclass` ¤

Everything needed to find a submitted run again.

Attributes:

Name	Type	Description
`schema_version`	`int`	The layout of this record.
`run_id`	`str`	The run's name, unique across backends; also its outputs' directory name.
`backend`	`str`	The name of the backend that ran it.
`job`	`str`	The job's name.
`project`	`str`	The project root it was submitted from.
`submitted_at`	`datetime`	When it was submitted, in UTC.
`settings`	`Mapping[str, JsonValue]`	Frozen provider configuration used for every recovery operation.
`details`	`Mapping[str, str]`	The provider's own references to the run, such as an app or cluster id.

`NewRun(*, run_id, spec, project, settings)` `dataclass` ¤

A run being submitted: its id, its job and its project, until the provider names it.

Attributes:

Name	Type	Description
`run_id`	`str`	The run's id, from :func:`new_run_id`.
`spec`	`JobSpec`	The job.
`project`	`Path`	The resolved project root.
`settings`	`Mapping[str, JsonValue]`	Frozen provider configuration used to construct the run handle.

`handle(backend, details)` ¤

The handle of this run, submitted now.

Parameters:

Name	Type	Description	Default
`backend`	`str`	The backend's name.	required
`details`	`Mapping[str, str]`	The provider's references to the run.	required

Returns:

Type	Description
`RunHandle`	The handle.

`ComputeBackend` ¤

Bases: Protocol

A compute provider that runs jobs.

Every method but :meth:submit takes a handle this backend returned, possibly in another process, and must work from it alone.

`name` `instance-attribute` ¤

The name the backend is registered under.

`submit(spec, *, project)` ¤

Start spec from the project at project and return once it is submitted.

`status(handle)` ¤

Where the run is now.

`logs(handle, *, follow)` ¤

The worker's output lines; with follow, until the run ends.

`fetch(handle, destination)` ¤

Copy the run's outputs to destination / handle.run_id and return that directory.

`cancel(handle)` ¤

Stop the run; a run that already ended is left as it is.

`RunStore(root=None)` ¤

The handles of submitted runs, one JSON file per run.

Keep handles under root.

Parameters:

Name	Type	Description	Default
`root`	`Path \| None`	The directory; `default_state_dir() / "compute" / "runs"` when `None`.	`None`

`save(handle)` ¤

Write a handle, replacing an earlier one of the same run.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The handle.	required

`load(run_id)` ¤

Read the handle of a run.

Parameters:

Name	Type	Description	Default
`run_id`	`str`	The run.	required

Returns:

Type	Description
`RunHandle`	Its handle.

Raises:

Type	Description
`KeyError`	If no handle of that run is kept; the message names the directory.

`read_all()` ¤

Every kept handle, newest first, and each record that cannot be read, with the reason.

A record an older substrax wrote, or a damaged one, is named on its own rather than hiding the runs that can still be read.

Returns:

Type	Description
`StoredRuns`	The readable handles and the unreadable records.

`StoredRuns(*, handles, unreadable)` `dataclass` ¤

What a :class:RunStore holds: the handles it can read, and why it cannot read the rest.

Attributes:

Name	Type	Description
`handles`	`tuple[RunHandle, ...]`	The readable handles, newest first.
`unreadable`	`Mapping[Path, str]`	Each record that cannot be read, by path, with the reason.

`new_run_id(job)` ¤

Name a new run of job by its UTC start and a random suffix.

Parameters:

Name	Type	Description	Default
`job`	`str`	The job's name.	required

Returns:

Type	Description
`str`	An id such as `examples-20260919T120000Z-3f9a1c`.

`default_state_dir()` ¤

The directory substrax keeps run handles and local runs in.

Returns:

Type	Description
`Path`	`$SUBSTRAX_STATE_DIR` when set, else `$XDG_STATE_HOME/substrax`, else
`Path`	`~/.local/state/substrax`.

`backend_state_dir(state_dir, backend)` ¤

The directory a backend keeps its runs' local files in.

Parameters:

Name	Type	Description	Default
`state_dir`	`str \| None`	The backend's `state_dir` setting; the default state directory when `None`.	required
`backend`	`str`	The backend's name.	required

Returns:

Type	Description
`Path`	`<state_dir>/compute/<backend>`.

`new_run(spec, project, *, settings)` ¤

Start submitting a run of spec from project.

Parameters:

Name	Type	Description	Default
`spec`	`JobSpec`	The job.	required
`project`	`Path`	The project root.	required
`settings`	`Mapping[str, JsonValue]`	Validated provider settings to freeze into the handle.	required

Returns:

Type	Description
`NewRun`	The run, named by :func:`new_run_id`.

`stage_run(runs, run)` ¤

Write the run's spec into a directory of its own under runs.

Parameters:

Name	Type	Description	Default
`runs`	`Path`	The backend's directory of runs.	required
`run`	`NewRun`	The run.	required

Returns:

Type	Description
`Path`	Its directory, which holds `spec.json`.

`substrax.compute.config` ¤

A project's compute configuration: the [tool.substrax.compute] table of its pyproject.toml.

[tool.substrax.compute]
backend = "modal"                        # used when the command line names none

[tool.substrax.compute.backends.modal]   # passed to the backend as its settings
outputs_volume = "demo-outputs"

[tool.substrax.compute.jobs.examples]
examples = ["examples/metrics"]          # one task per script, as discover_examples lists them
commands = { gpu-tests = ["python", "-m", "pytest", "tests/gpu"] }
extras = ["cuda12"]
resources = { accelerator = { kind = "L4", count = 1 }, cpu = 8, memory_mib = 32768 }
timeout_seconds = 1800
task_timeout_seconds = 300               # each task's budget; the job's when unset
runtime = { platforms = ["cuda"], xla_flags = ["--xla_gpu_deterministic_ops=true"] }
env = { TF_CPP_MIN_LOG_LEVEL = "1" }
mounts = [{ name = "datasets", path = "/data", access = "read-only" }]

A misspelled key is refused rather than ignored.

`TaskSources(*, timeout_seconds, examples=(), commands=dict(), task_timeout_seconds=None)` `dataclass` ¤

Where a configured job's tasks come from, and their budget.

The job's other keys are :class:~substrax.compute.JobSpec fields.

Attributes:

Name	Type	Description
`timeout_seconds`	`float`	The job's budget, which is also a `JobSpec` field.
`examples`	`tuple[str, ...]`	Directories, relative to the project, whose example scripts become tasks.
`commands`	`Mapping[str, tuple[str, ...]]`	Further tasks, by name, run after the examples.
`task_timeout_seconds`	`float \| None`	Each task's budget; `timeout_seconds` when `None`.

`ComputeConfig(*, backend=None, backends=dict(), jobs=dict())` `dataclass` ¤

A project's compute configuration.

Attributes:

Name	Type	Description
`backend`	`str \| None`	The backend used when none is named; `None` requires naming one.
`backends`	`Mapping[str, Mapping[str, JsonValue]]`	Each backend's settings, by backend name.
`jobs`	`Mapping[str, Mapping[str, JsonValue]]`	Each job's table, by name, read into a `JobSpec` when the job is resolved.

`read_compute_config(project)` ¤

Read the [tool.substrax.compute] table of the project's pyproject.toml.

Parameters:

Name	Type	Description	Default
`project`	`Path`	The project root.	required

Returns:

Type	Description
`ComputeConfig`	The configuration.

Raises:

Type	Description
`LookupError`	If pyproject.toml has no such table.
`ValueError`	If a key is unknown or a value is of the wrong type.

`resolve_job(config, name, *, project, accelerator=None)` ¤

Build the spec of a configured job, listing its examples now.

The job's table holds examples, commands and task_timeout_seconds, which make its tasks, and otherwise the fields of :class:~substrax.compute.JobSpec under their own names.

Parameters:

Name	Type	Description	Default
`config`	`ComputeConfig`	The project's configuration.	required
`name`	`str`	The job.	required
`project`	`Path`	The project root.	required
`accelerator`	`Accelerator \| None`	Replaces the configured accelerator for this run.	`None`

Returns:

Type	Description
`JobSpec`	The spec.

Raises:

Type	Description
`KeyError`	If the job is not configured; the message lists the configured ones.
`FileNotFoundError`	If an examples directory does not exist.
`ValueError`	If a key of the job's table is unknown or a value of the wrong type.

`parse_accelerator(text)` ¤

Read an accelerator written as KIND or KIND:COUNT, such as H100:8.

Parameters:

Name	Type	Description	Default
`text`	`str`	The accelerator.	required

Returns:

Type	Description
`Accelerator`	It.

Raises:

Type	Description
`ValueError`	If `text` is not of that form.

`require_locked_substrax(project)` ¤

Return the substrax the project locks, which the worker runs from.

Parameters:

Name	Type	Description	Default
`project`	`Path`	The project root.	required

Returns:

Type	Description
`str`	The locked version, or `"editable"` when the project is substrax or installs it from a
`str`	path.

Raises:

Type	Description
`FileNotFoundError`	If the project has no uv.lock.
`LookupError`	If its uv.lock does not include substrax.

`substrax.compute.worker` ¤

The program every backend runs on the remote machine: a job's tasks, their logs and a manifest.

Run as python -m substrax.compute.worker --spec spec.json --project . --outputs /outputs from the project's environment. Each task runs from the project root with AVITAI_OUTPUT_DIR set to <outputs>/<task>, so whatever it writes through :func:~substrax.artifacts.resolve_output_dir lands beside its stdout.log and stderr.log. Before the first task a child with the job's JAX runtime reports the devices the run got, which the manifest records. The worker writes <outputs>/manifest.json after every task, so a run cut short still says what finished. Its exit code is 0 when every task succeeded and 1 otherwise.

`DEVICES_LOG = 'devices.log'` `module-attribute` ¤

The device probe's output when it fails, under the run's outputs.

`TaskStatus` ¤

Bases: StrEnum

How a task ended.

`TaskOutcome(*, name, argv, status, returncode, seconds, outputs)` `dataclass` ¤

What one task did.

Attributes:

Name	Type	Description
`name`	`str`	The task.
`argv`	`tuple[str, ...]`	The command that ran, with `python` resolved to the interpreter.
`status`	`TaskStatus`	How it ended.
`returncode`	`int \| None`	Its exit code; `None` when it could not start.
`seconds`	`float`	Wall-clock seconds from start to end.
`outputs`	`tuple[str, ...]`	The files it wrote under its directory, relative to it, without the logs.

`RunDevices(*, platform, kinds)` `dataclass` ¤

The devices a run's JAX saw, which the provider allocated for the requested ones.

Attributes:

Name	Type	Description
`platform`	`str`	The default backend, as JAX names it (`"gpu"`, `"cpu"`, `"tpu"`).
`kinds`	`tuple[str, ...]`	One `device_kind` per device, in JAX's order, such as `"NVIDIA H200"`.

`JobManifest(*, schema_version=MANIFEST_VERSION, job, tasks=(), finished=False, devices=None)` `dataclass` ¤

What a run of a job did, task by task.

Attributes:

Name	Type	Description
`schema_version`	`int`	The layout of this record.
`job`	`str`	The job's name.
`tasks`	`tuple[TaskOutcome, ...]`	The tasks that ran, in order.
`finished`	`bool`	Whether the worker ran every task; a run cut short leaves it `False`.
`devices`	`RunDevices \| None`	The devices the run's JAX saw; `None` when the probe failed, with its output in `devices.log`.

`succeeded` `property` ¤

Whether every task succeeded.

`read_manifest(data)` ¤

Read a manifest from its parsed JSON object.

Parameters:

Name	Type	Description	Default
`data`	`dict[str, JsonValue]`	The JSON object the worker wrote.	required

Returns:

Type	Description
`JobManifest`	The manifest.

Raises:

Type	Description
`ValueError`	If the manifest's version is not this release's, or a field does not match its annotation.

`run_job(spec, *, project, outputs, echo=True, python=sys.executable)` ¤

Run every task of spec in order and write the manifest after each.

Each task runs for at most its own budget or what is left of the job's, whichever is less; a task the job's budget leaves no time for is recorded as timed out without starting.

Parameters:

Name	Type	Description	Default
`spec`	`JobSpec`	The job.	required
`project`	`Path`	The project root each task runs from.	required
`outputs`	`Path`	The directory the run's logs, outputs and manifest go to.	required
`echo`	`bool`	Also copy each line a task writes to this process's standard output or error, prefixed with the task's name, so the provider's log shows the run as it goes.	`True`
`python`	`str`	The interpreter a task's leading `"python"` names.	`executable`

Returns:

Type	Description
`JobManifest`	The manifest.

`main(argv=None)` ¤

Run the job a spec file describes.

Parameters:

Name	Type	Description	Default
`argv`	`Sequence[str] \| None`	The command-line arguments; this process's when `None`.	`None`

Returns:

Type	Description
`int`	0 when every task succeeded, 1 otherwise.

`substrax.compute.registry` ¤

Backends by name, from the substrax.compute.backends entry-point group.

substrax registers its own backends (local, modal, skypilot) in that group, and any other distribution adds a provider by registering a :data:~substrax.compute.backend.BackendFactory there, so a built-in backend and a third-party one load the same way. A backend's module is imported only when its name is chosen; one whose provider library is missing raises the ImportError its module raises, naming the extra to install.

`UnknownBackendError` ¤

Bases: LookupError

No registered backend has the requested name.

`backend_names(*, discover=entry_points)` ¤

The names of the registered backends, sorted.

Parameters:

Name	Type	Description	Default
`discover`	`Callable[..., EntryPoints]`	Lists entry points by group; :func:`importlib.metadata.entry_points` by default.	`entry_points`

Returns:

Type	Description
`tuple[str, ...]`	The names.

`load_backend(name, settings=None, *, discover=entry_points)` ¤

Build the backend registered under name from its settings.

Parameters:

Name	Type	Description	Default
`name`	`str`	The backend's name.	required
`settings`	`Mapping[str, JsonValue] \| None`	Its settings table; empty when `None`.	`None`
`discover`	`Callable[..., EntryPoints]`	Lists entry points by group; :func:`importlib.metadata.entry_points` by default.	`entry_points`

Returns:

Type	Description
`ComputeBackend`	The backend.

Raises:

Type	Description
`UnknownBackendError`	If no backend has that name; the message lists the registered ones.
`TypeError`	If the entry point builds something that is not a backend.

`substrax.compute.local` ¤

The local backend: runs a job's worker on this machine, detached from the shell.

It runs the project's current environment as it is (uv run --no-sync), so a job's extras and accelerator are not provisioned here: the machine has what it has. It is the reference backend the contract tests run against, and it serves a developer with a local accelerator.

Settings ([tool.substrax.compute.backends.local]):

python: the interpreter command that runs the worker, such as ".venv/bin/python"; uv run --no-sync --project <project> python when unset.
state_dir: where runs are kept; :func:~substrax.compute.backend.default_state_dir when unset.

`LocalSettings(*, python=None, state_dir=None)` `dataclass` ¤

The local backend's settings.

Attributes:

Name	Type	Description
`python`	`str \| None`	The interpreter command that runs the worker; the project's through `uv` when `None`.
`state_dir`	`str \| None`	Where runs are kept; the default state directory when `None`.

`LocalBackend(settings=None)` ¤

Runs jobs on this machine.

Read the backend's settings.

Parameters:

Name	Type	Description	Default
`settings`	`Mapping[str, JsonValue] \| None`	The settings table; every setting has a default.	`None`

Raises:

Type	Description
`ValueError`	If a setting is unknown or of the wrong type.

`submit(spec, *, project)` ¤

Start the worker on spec in the background and return its handle.

Parameters:

Name	Type	Description	Default
`spec`	`JobSpec`	The job.	required
`project`	`Path`	The project root.	required

Returns:

Type	Description
`RunHandle`	The run's handle.

`status(handle)` ¤

Where the run is: from its manifest when it finished, else from its process.

The worker's liveness is observed before the manifest is read: a worker that was gone wrote its last manifest before exiting, so the read that follows sees it. Read the other way round, a run finishing between the two checks would look unfinished and gone.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required

Returns:

Type	Description
`RunState`	Its state.

`logs(handle, *, follow)` ¤

The worker's output lines, including every task's, prefixed with the task's name.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required
`follow`	`bool`	Keep reading until the run ends.	required

Yields:

Name	Type	Description
`str`	`str`	Each line, without its line ending.

`fetch(handle, destination)` ¤

Copy the run's outputs to destination / handle.run_id.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required
`destination`	`Path`	The directory to copy into.	required

Returns:

Type	Description
`Path`	The copy.

`cancel(handle)` ¤

Stop the worker and its tasks; a run that already ended is left as it is.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required

`substrax.compute.modal_backend` ¤

The modal backend: runs a job on Modal (https://modal.com); needs substrax[modal].

The image is built in layers keyed on what they depend on, so editing the project's source rebuilds only the last two:

Image.uv_sync installs the locked dependencies and the job's extras, from pyproject.toml and uv.lock alone, on this interpreter's Python version, which the serialized worker needs;
the project's source is copied in;
the project itself is installed into that environment without dependencies.

Image.uv_sync needs a Modal image builder of 2025.06 or later, which is a workspace setting.

The worker runs in a Modal function, detached from the submitting shell, with the outputs volume mounted at /outputs and each of the job's mounts at its path. Logs come from modal app logs, the provider's own command, run by this interpreter.

Settings ([tool.substrax.compute.backends.modal]):

outputs_volume (required): the Modal Volume the runs' outputs go to, created when missing.
volumes: the Modal Volume behind each of a job's mounts, by mount name.
environment: the Modal environment; the profile's default when unset.
ignore: patterns of project files left out of the image, .dockerignore style.
state_dir: where cancellations are recorded; the default state directory when unset.

`ModalSettings(*, outputs_volume, volumes=dict(), environment=None, ignore=_DEFAULT_IGNORE, state_dir=None)` `dataclass` ¤

The modal backend's settings; see the module documentation.

Attributes:

Name	Type	Description
`outputs_volume`	`str`	The Volume the runs' outputs go to.
`volumes`	`Mapping[str, str]`	The Volume behind each mount, by mount name.
`environment`	`str \| None`	The Modal environment; the profile's default when `None`.
`ignore`	`tuple[str, ...]`	Project files left out of the image.
`state_dir`	`str \| None`	Where cancellations are recorded.

`ModalBackend(settings)` ¤

Runs jobs on Modal.

Read the backend's settings.

Parameters:

Name	Type	Description	Default
`settings`	`Mapping[str, JsonValue]`	The settings table; `outputs_volume` is required.	required

Raises:

Type	Description
`ValueError`	If a setting is missing, unknown or of the wrong type.

`submit(spec, *, project)` ¤

Build the image and spawn the worker in a detached app.

Parameters:

Name	Type	Description	Default
`spec`	`JobSpec`	The job.	required
`project`	`Path`	The project root.	required

Returns:

Type	Description
`RunHandle`	The run's handle.

`status(handle)` ¤

Where the run is: running while its call has no result, then from its manifest.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required

Returns:

Type	Description
`RunState`	Its state.

`logs(handle, *, follow)` ¤

The app's log lines, from modal app logs.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required
`follow`	`bool`	Keep reading until the run ends.	required

Yields:

Name	Type	Description
`str`	`str`	Each line, without its line ending.

`fetch(handle, destination)` ¤

Copy the run's directory of the outputs volume to destination / handle.run_id.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required
`destination`	`Path`	The directory to copy into.	required

Returns:

Type	Description
`Path`	The copy.

`cancel(handle)` ¤

Cancel the call and stop its container; a run that already ended is left as it is.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required

`run_worker(spec_json, run_id, project, outputs, volume)` ¤

Run the job inside the Modal function and commit its outputs.

Parameters:

Name	Type	Description	Default
`spec_json`	`str`	The job spec as JSON text.	required
`run_id`	`str`	The run, whose directory under `outputs` the files go to.	required
`project`	`str`	The project's path in the container.	required
`outputs`	`str`	The outputs volume's path in the container.	required
`volume`	`str`	The outputs volume's name.	required

Returns:

Type	Description
`bool`	Whether every task succeeded.

`substrax.compute.skypilot_backend` ¤

The skypilot backend: runs a job through SkyPilot (https://docs.skypilot.ai) on its clouds.

SkyPilot is a separate tool, installed the way its documentation says, in an environment of its own (uv tool install --with pip "skypilot[gcp]"). The scientific environment never imports it: every operation runs substrax/compute/sky_sdk.py in SkyPilot's own interpreter, which calls SkyPilot's public SDK and reads its native request and queue records.

Each run gets a cluster of its own. The task syncs the project as its workdir, installs the locked environment (uv sync --frozen) and runs the worker. Before it waits for provisioning, a launch records the run's intent and SkyPilot's request id beside the task file, so an interrupted launch is reconciled from them rather than submitted again. Cancelling cancels that request and the job, then tears the cluster down, keeping every failure.

How inputs reach the cluster and outputs come back is the transport's, configured as exactly one of [...skypilot.bucket] or [...skypilot.copy]; see :mod:substrax.compute.skypilot_transports.

Settings ([tool.substrax.compute.backends.skypilot]):

bucket or copy: the transport's table.
setup_commands: commands that prepare the image before the locked environment is synced.
infra: where to run, in SkyPilot's infra syntax, such as gcp or runpod; any enabled cloud when unset.
use_spot: run on spot instances.
sky: the sky command, used to find SkyPilot's interpreter; sky on PATH.
sky_python: SkyPilot's interpreter, when it is not beside the sky command.
worker_python: the command that starts Python in the synced environment; uv run --no-sync python by default.
state_dir: where task files, launch records and cancellations are kept; the default state directory when unset.

`SkyPilotSettings(*, bucket=None, copy=None, setup_commands=(), infra=None, use_spot=False, sky='sky', sky_python=None, worker_python='uv run --no-sync python', state_dir=None)` `dataclass` ¤

The skypilot backend's settings; see the module documentation.

Attributes:

Name	Type	Description
`bucket`	`BucketSettings \| None`	Bucket transport's settings; set exactly one of `bucket` and `copy`.
`copy`	`CopySettings \| None`	Copy transport's settings.
`setup_commands`	`tuple[str, ...]`	Commands run before the locked environment is synced.
`infra`	`str \| None`	Where to run, in SkyPilot's `infra` syntax.
`use_spot`	`bool`	Run on spot instances.
`sky`	`str`	The `sky` command, used to find SkyPilot's interpreter.
`sky_python`	`str \| None`	SkyPilot's interpreter; the `python` beside `sky` when unset.
`worker_python`	`str`	The command that starts Python in the synced environment.
`state_dir`	`str \| None`	Where task files, launch records and cancellations are kept.

`SkyPilotBackend(settings)` ¤

Runs jobs through SkyPilot's public SDK, in SkyPilot's own interpreter.

Read the backend's settings.

Parameters:

Name	Type	Description	Default
`settings`	`Mapping[str, JsonValue]`	The settings table, with exactly one of `bucket` and `copy`.	required

Raises:

Type	Description
`ValueError`	If a setting is missing, unknown or of the wrong type, or the transport is not exactly one of `bucket` and `copy`, or its own checks fail.

`submit(spec, *, project)` ¤

Write the task file, record the launch, and launch it on a cluster of its own.

Parameters:

Name	Type	Description	Default
`spec`	`JobSpec`	The job.	required
`project`	`Path`	The project root.	required

Returns:

Type	Description
`RunHandle`	The run's handle, once the job is submitted on the cluster.

`status(handle)` ¤

The job's state from the cluster's queue while it is up, else from its outputs.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required

Returns:

Type	Description
`RunState`	Its state.

Raises:

Type	Description
`RuntimeError`	If the queue cannot be read and the outputs cannot answer for the run.
`ValueError`	If SkyPilot reports a job status this backend does not know.

`logs(handle, *, follow)` ¤

The job's output lines, from SkyPilot's log stream.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required
`follow`	`bool`	Keep reading until the job ends.	required

Yields:

Name	Type	Description
`str`	`str`	Each line, without its line ending.

Raises:

Type	Description
`RuntimeError`	If the log stream fails.

`fetch(handle, destination)` ¤

Copy the run's outputs to destination / handle.run_id.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required
`destination`	`Path`	The directory to copy into.	required

Returns:

Type	Description
`Path`	The copy.

`cancel(handle)` ¤

Cancel unfinished work and tear its cluster down; a run that already ended is left.

Every step is attempted, whatever the one before it did.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required

Raises:

Type	Description
`ExceptionGroup`	If any step failed; it holds each failure.

`teardown(handle)` ¤

Remove the run's cluster, and with it anything left on it.

Unlike :meth:cancel, this removes the cluster of a run that already ended too; fetch a copy transport's outputs first.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required

`task_for(spec, *, project, run_id, run_dir)` ¤

The SkyPilot task that runs spec, as its task file holds it.

Parameters:

Name	Type	Description	Default
`spec`	`JobSpec`	The job.	required
`project`	`Path`	The project root, synced as the task's workdir.	required
`run_id`	`str`	The run.	required
`run_dir`	`Path`	The run's staged directory, which holds its spec.	required

Returns:

Type	Description
`dict[str, JsonValue]`	The task.

`substrax.compute.skypilot_transports` ¤

How a SkyPilot run's inputs reach its cluster and its outputs come back.

The skypilot backend runs every job the same way and leaves this to its transport, one of:

:class:BucketTransport: inputs and outputs are object-store buckets SkyPilot mounts (mode: MOUNT), so the outputs outlive the cluster, which SkyPilot tears down when the job ends. Outputs are fetched with the bucket's own tool: gcloud storage for gs://, aws s3 for s3://.
:class:CopyTransport: SkyPilot copies explicit local inputs onto the cluster, read-only, and the outputs are fetched over SSH and rsync, verified against complete manifests of the remote directory taken before and after the copy. The cluster has to stay up until then, so it is removed by an explicit teardown, or by its autostop timer, which starts after setup and is not a provisioning deadline or a cost cap. The worker runs under bubblewrap, which the image has to provide with working user namespaces. Providers without bucket mounts (RunPod, Vast) use it.

`BucketSettings(*, outputs_bucket, buckets=dict())` `dataclass` ¤

[tool.substrax.compute.backends.skypilot.bucket]: buckets SkyPilot mounts.

Attributes:

Name	Type	Description
`outputs_bucket`	`str`	The `gs://` or `s3://` URI the runs' outputs go to.
`buckets`	`Mapping[str, str]`	The bucket URI behind each of a job's mounts, by mount name.

`CopySettings(*, sources=dict(), disk_size=60, lifetime_minutes=60, network_isolation=False)` `dataclass` ¤

[tool.substrax.compute.backends.skypilot.copy]: local inputs copied to the cluster.

Attributes:

Name	Type	Description
`sources`	`Mapping[str, str]`	The local path behind each of a job's mounts, by mount name; read-only.
`disk_size`	`int`	The disk the cluster is asked for, in GB.
`lifetime_minutes`	`int`	How long the cluster may idle after setup before it removes itself.
`network_isolation`	`bool`	Run the worker without network access.

`Transport` ¤

Bases: Protocol

The part of a SkyPilot run that depends on how its files move.

`down_with_job` `property` ¤

Whether SkyPilot removes the cluster as soon as its job ends.

`recovers_without_queue` `property` ¤

Whether the outputs answer for the run when its cluster's queue cannot be read.

`add_to_task(spec, resources, mounts)` ¤

Add the transport's resources and the job's input and output mounts to the task.

`wrap_worker(worker, spec)` ¤

The command that runs worker on the cluster.

`fetch(handle, target)` ¤

Copy the run's outputs into target.

`state_from_outputs(handle)` ¤

The run's state once its cluster no longer reports it, from what it left behind.

`BucketTransport(settings, *, infra)` ¤

Inputs and outputs on buckets SkyPilot mounts; see the module documentation.

Check the settings against where the run goes.

Parameters:

Name	Type	Description	Default
`settings`	`BucketSettings`	The bucket settings.	required
`infra`	`str \| None`	The backend's `infra`.	required

Raises:

Type	Description
`ValueError`	If the provider cannot mount buckets, or the outputs bucket's scheme has no download tool.

`add_to_task(spec, resources, mounts)` ¤

Mount the outputs bucket and each input mount's bucket.

Parameters:

Name	Type	Description	Default
`spec`	`JobSpec`	The job.	required
`resources`	`dict[str, JsonValue]`	The task's resources; unchanged.	required
`mounts`	`dict[str, JsonValue]`	The task's file mounts.	required

Raises:

Type	Description
`LookupError`	If a mount has no configured bucket.

`wrap_worker(worker, spec)` ¤

The worker runs as it is.

Parameters:

Name	Type	Description	Default
`worker`	`str`	The worker command.	required
`spec`	`JobSpec`	The job.	required

Returns:

Type	Description
`str`	`worker`.

`fetch(handle, target)` ¤

Download the run's directory of the outputs bucket.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required
`target`	`Path`	The directory to copy into.	required

`state_from_outputs(handle)` ¤

The state the run's manifest in the outputs bucket records.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required

Returns:

Type	Description
`RunState`	Its state; `failed` when the run left no finished manifest.

`CopyTransport(settings, *, worker_python, runs)` ¤

Local inputs copied to the cluster, outputs fetched over SSH; see the module documentation.

Check the settings.

Parameters:

Name	Type	Description	Default
`settings`	`CopySettings`	The copy settings.	required
`worker_python`	`str`	The command that starts Python in the cluster's synced environment.	required
`runs`	`Path`	Where the backend keeps its runs' local files.	required

Raises:

Type	Description
`ValueError`	If the disk size or the lifetime is not positive.

`add_to_task(spec, resources, mounts)` ¤

Ask for the disk and the autostop timer, and copy each input mount's local source.

Parameters:

Name	Type	Description	Default
`spec`	`JobSpec`	The job.	required
`resources`	`dict[str, JsonValue]`	The task's resources.	required
`mounts`	`dict[str, JsonValue]`	The task's file mounts.	required

Raises:

Type	Description
`ValueError`	If a mount is writable; a copy cannot be written back.
`LookupError`	If a mount has no configured source.

`wrap_worker(worker, spec)` ¤

Run the worker under bubblewrap: the system, the project and the inputs read-only.

Parameters:

Name	Type	Description	Default
`worker`	`str`	The worker command.	required
`spec`	`JobSpec`	The job; each mount is bound read-only.	required

Returns:

Type	Description
`str`	The bubblewrap command around `worker`.

`fetch(handle, target)` ¤

Copy the outputs over rsync and verify them against the remote directory's manifest.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required
`target`	`Path`	The directory to copy into.	required

Raises:

Type	Description
`ValueError`	If the remote outputs changed during the copy, or the copy differs from them; the cluster is left up.

`state_from_outputs(handle)` ¤

The state of the run's verified local copy of its outputs.

Parameters:

Name	Type	Description	Default
`handle`	`RunHandle`	The run.	required

Returns:

Type	Description
`RunState`	Its state.

Raises:

Type	Description
`RuntimeError`	If no verified copy exists, or it records an unfinished job.

`run_tool(command)` ¤

Run a command line tool and return its standard output.

Parameters:

Name	Type	Description	Default
`command`	`Sequence[str]`	The command.	required

Returns:

Type	Description
`str`	Its standard output.

Raises:

Type	Description
`RuntimeError`	If it exits non-zero; the message holds its standard error.

`downloader(uri)` ¤

The command that downloads a bucket URI's directory.

Parameters:

Name	Type	Description	Default
`uri`	`str`	A `gs://` or `s3://` URI.	required

Returns:

Type	Description
`tuple[str, ...]`	The command, to which the source and the destination are appended.

Raises:

Type	Description
`ValueError`	If the scheme has no download tool.

Compute¤

Declaring jobs¤

Running them¤

What a run produces¤

Backends¤

SkyPilot¤

Run handles¤

Adding a provider¤

substrax.compute ¤

JOB_SPEC_VERSION = 2 module-attribute ¤

Accelerator(*, kind, count=1) dataclass ¤

__post_init__() ¤

JobSpec(*, schema_version=JOB_SPEC_VERSION, name, tasks, timeout_seconds, resources=Resources(), extras=(), runtime=JaxRuntime(), env=dict(), mounts=()) dataclass ¤

__post_init__() ¤

Mount(*, name, path, access) dataclass ¤

__post_init__() ¤

MountAccess ¤

Resources(*, accelerator=None, cpu=None, memory_mib=None) dataclass ¤

__post_init__() ¤

Task(*, name, argv, timeout_seconds) dataclass ¤

__post_init__() ¤

example_tasks(paths, *, project, timeout_seconds) ¤

read_job_spec(data) ¤

substrax.compute.backend ¤

SPEC_NAME = 'spec.json' module-attribute ¤

STATE_DIR_ENV = 'SUBSTRAX_STATE_DIR' module-attribute ¤

BackendFactory = Callable[[Mapping[str, JsonValue]], ComputeBackend] module-attribute ¤

RunState ¤

is_final property ¤

RunHandle(*, schema_version=HANDLE_VERSION, run_id, backend, job, project, submitted_at, settings, details=dict()) dataclass ¤

NewRun(*, run_id, spec, project, settings) dataclass ¤

handle(backend, details) ¤

ComputeBackend ¤

name instance-attribute ¤

submit(spec, *, project) ¤

status(handle) ¤

logs(handle, *, follow) ¤

fetch(handle, destination) ¤

cancel(handle) ¤

RunStore(root=None) ¤

save(handle) ¤

load(run_id) ¤

read_all() ¤

StoredRuns(*, handles, unreadable) dataclass ¤

new_run_id(job) ¤

default_state_dir() ¤

backend_state_dir(state_dir, backend) ¤

new_run(spec, project, *, settings) ¤

stage_run(runs, run) ¤

substrax.compute.config ¤

TaskSources(*, timeout_seconds, examples=(), commands=dict(), task_timeout_seconds=None) dataclass ¤

ComputeConfig(*, backend=None, backends=dict(), jobs=dict()) dataclass ¤

read_compute_config(project) ¤

resolve_job(config, name, *, project, accelerator=None) ¤

parse_accelerator(text) ¤

require_locked_substrax(project) ¤

substrax.compute.worker ¤

DEVICES_LOG = 'devices.log' module-attribute ¤

TaskStatus ¤

TaskOutcome(*, name, argv, status, returncode, seconds, outputs) dataclass ¤

RunDevices(*, platform, kinds) dataclass ¤

JobManifest(*, schema_version=MANIFEST_VERSION, job, tasks=(), finished=False, devices=None) dataclass ¤

succeeded property ¤

read_manifest(data) ¤

run_job(spec, *, project, outputs, echo=True, python=sys.executable) ¤

main(argv=None) ¤

substrax.compute.registry ¤

UnknownBackendError ¤

backend_names(*, discover=entry_points) ¤

load_backend(name, settings=None, *, discover=entry_points) ¤

substrax.compute.local ¤

LocalSettings(*, python=None, state_dir=None) dataclass ¤

LocalBackend(settings=None) ¤

submit(spec, *, project) ¤

status(handle) ¤

logs(handle, *, follow) ¤

fetch(handle, destination) ¤

cancel(handle) ¤

substrax.compute.modal_backend ¤

ModalSettings(*, outputs_volume, volumes=dict(), environment=None, ignore=_DEFAULT_IGNORE, state_dir=None) dataclass ¤

`substrax.compute` ¤

`JOB_SPEC_VERSION = 2` `module-attribute` ¤

`Accelerator(*, kind, count=1)` `dataclass` ¤

`__post_init__()` ¤

`JobSpec(*, schema_version=JOB_SPEC_VERSION, name, tasks, timeout_seconds, resources=Resources(), extras=(), runtime=JaxRuntime(), env=dict(), mounts=())` `dataclass` ¤

`__post_init__()` ¤

`Mount(*, name, path, access)` `dataclass` ¤

`__post_init__()` ¤

`MountAccess` ¤

`Resources(*, accelerator=None, cpu=None, memory_mib=None)` `dataclass` ¤

`__post_init__()` ¤

`Task(*, name, argv, timeout_seconds)` `dataclass` ¤

`__post_init__()` ¤

`example_tasks(paths, *, project, timeout_seconds)` ¤

`read_job_spec(data)` ¤

`substrax.compute.backend` ¤

`SPEC_NAME = 'spec.json'` `module-attribute` ¤

`STATE_DIR_ENV = 'SUBSTRAX_STATE_DIR'` `module-attribute` ¤

`BackendFactory = Callable[[Mapping[str, JsonValue]], ComputeBackend]` `module-attribute` ¤

`RunState` ¤

`is_final` `property` ¤

`RunHandle(*, schema_version=HANDLE_VERSION, run_id, backend, job, project, submitted_at, settings, details=dict())` `dataclass` ¤

`NewRun(*, run_id, spec, project, settings)` `dataclass` ¤

`handle(backend, details)` ¤

`ComputeBackend` ¤

`name` `instance-attribute` ¤

`submit(spec, *, project)` ¤

`status(handle)` ¤

`logs(handle, *, follow)` ¤

`fetch(handle, destination)` ¤

`cancel(handle)` ¤

`RunStore(root=None)` ¤

`save(handle)` ¤

`load(run_id)` ¤

`read_all()` ¤

`StoredRuns(*, handles, unreadable)` `dataclass` ¤

`new_run_id(job)` ¤

`default_state_dir()` ¤

`backend_state_dir(state_dir, backend)` ¤

`new_run(spec, project, *, settings)` ¤

`stage_run(runs, run)` ¤

`substrax.compute.config` ¤

`TaskSources(*, timeout_seconds, examples=(), commands=dict(), task_timeout_seconds=None)` `dataclass` ¤

`ComputeConfig(*, backend=None, backends=dict(), jobs=dict())` `dataclass` ¤

`read_compute_config(project)` ¤

`resolve_job(config, name, *, project, accelerator=None)` ¤

`parse_accelerator(text)` ¤

`require_locked_substrax(project)` ¤

`substrax.compute.worker` ¤

`DEVICES_LOG = 'devices.log'` `module-attribute` ¤

`TaskStatus` ¤

`TaskOutcome(*, name, argv, status, returncode, seconds, outputs)` `dataclass` ¤

`RunDevices(*, platform, kinds)` `dataclass` ¤

`JobManifest(*, schema_version=MANIFEST_VERSION, job, tasks=(), finished=False, devices=None)` `dataclass` ¤

`succeeded` `property` ¤

`read_manifest(data)` ¤

`run_job(spec, *, project, outputs, echo=True, python=sys.executable)` ¤

`main(argv=None)` ¤

`substrax.compute.registry` ¤

`UnknownBackendError` ¤

`backend_names(*, discover=entry_points)` ¤

`load_backend(name, settings=None, *, discover=entry_points)` ¤

`substrax.compute.local` ¤

`LocalSettings(*, python=None, state_dir=None)` `dataclass` ¤

`LocalBackend(settings=None)` ¤

`submit(spec, *, project)` ¤

`status(handle)` ¤

`logs(handle, *, follow)` ¤

`fetch(handle, destination)` ¤

`cancel(handle)` ¤

`substrax.compute.modal_backend` ¤

`ModalSettings(*, outputs_volume, volumes=dict(), environment=None, ignore=_DEFAULT_IGNORE, state_dir=None)` `dataclass` ¤

`ModalBackend(settings)` ¤

`submit(spec, *, project)` ¤

`status(handle)` ¤

`logs(handle, *, follow)` ¤

`fetch(handle, destination)` ¤

`cancel(handle)` ¤

`run_worker(spec_json, run_id, project, outputs, volume)` ¤

`substrax.compute.skypilot_backend` ¤