Local Docker execution
compute_horde_sdk.v1.local runs Linux containers on the configured Docker host.
Install with pip install "compute-horde-sdk[local]". It requires a running
Docker daemon accessible to the caller. Before every run the executor checks the
local image cache and pulls the image if it is missing (or does not match the
requested platform).
Existing matching images are reused without contacting the registry;
mutable tags are not refreshed. References without a tag use latest.
Pulls use registry credentials from the caller’s Docker configuration, including
configured credential helpers. Pull failures are reported as LocalJobLaunchError.
Docker connection
The client uses docker.from_env(); connection selection and API version
negotiation are handled by the Docker SDK.
Submit, wait, and stop
from pathlib import Path
from compute_horde_sdk.v1.local import (
LocalDockerJobSpec, LocalBindVolume, LocalDockerClient,
)
async def run_job():
client = LocalDockerClient()
job = await client.create_job(
LocalDockerJobSpec(
docker_image="busybox:1.37", # pulled automatically if missing
args=["sh", "-c", 'cat /input/ref.fa; printf "done\\n" >&2'],
volumes=[
LocalBindVolume(source=Path("/data/reference"), destination="/input"),
LocalBindVolume(source=Path("/data/results"), destination="/work", read_only=False),
],
env={"EXAMPLE": "value with spaces"},
cpus=2,
memory="4g",
user="1000:1000", # omit to retain the image default
platform="linux/amd64", # omit to retain Docker's default
execution_time_limit_sec=120,
),
stdout=Path("/data/results/output.vcf"),
)
print(job.uuid, job.container_name) # available immediately
try:
result = await job.wait()
completed = result.to_completed_process()
completed.check_returncode() # optional nonzero-exit exception
return completed
finally:
await job.stop() # repeatable, including after exit
create_job returns before container creation finishes. container_id
becomes available after creation; uuid and container_name are available
earlier. A custom container_name is supported. Docker containers carry the
io.computehorde.sdk.job label with the job UUID. A name collision fails
launch and never removes the other container.
Arguments are a sequence of individual strings. The SDK preserves the image’s ENTRYPOINT and passes arguments as Docker’s Cmd array. An empty sequence leaves the image’s CMD intact. Shell syntax is interpreted only when the caller explicitly runs a shell, as in the example. No shell is involved in stopping.
Bind volumes and execution policy
LocalBindVolume is an immutable, validated volume object with source,
destination, and read_only=True. Sources must already be regular files
or directories. Relative sources become absolute at construction. Destinations
are absolute Linux container paths, without the remote API’s /volume/ or
/output/ prefix restrictions. The volume list preserves repeated sources,
distinct destinations, and mixed modes. Docker rejects conflicting destinations.
The original host paths are bound directly. A writable mount may serve as both input and output. Bind paths refer to the Docker daemon’s host; when using a remote daemon, the same paths must also exist there. Source validation checks the caller’s filesystem.
cpus sets a per-container CPU quota (Docker NanoCpus, equivalent to
--cpus). memory sets a per-container byte limit, accepting positive bytes
or Docker sizes such as 512m and 4g. Unset CPU, memory, user, and platform
options retain Docker or image defaults.
Containers use Docker’s default networking.
The SDK attaches to stdout/stderr before starting the container and removes the container after collecting its output and exit status. Containers inherit the Docker daemon’s logging configuration. The SDK also captures stdout and stderr, optionally streaming stdout to the supplied destination. On explicit job cancellation, timeout, or launch failure, the SDK attempts to force-remove any owned container, including one that never started. Cleanup failures are reported to the caller. Bind-mounted files are retained.
Cancelling a task awaiting job.wait() does not stop the container; use
job.stop() or job.request_stop() to cancel the job.
There is no stdin forwarding: stdin is closed.
Output and results
Without stdout=, stdout and stderr are captured in full in memory. With
stdout= set to a path or open file object, stdout is written incrementally
with backpressure and is not accumulated in the result; result.stdout
is None. Stderr is always captured separately in full. Large stdout artifacts
should use a file; large stderr still consumes memory.
The default is bytes. text=True decodes captured streams with encoding
(default UTF-8) and errors (default strict). Text mode does not normalize
newlines. An open stdout file must accept bytes when text=False or strings
when text=True; incremental decoding preserves multibyte characters across
chunks. A path destination always receives exact original bytes. The SDK opens
paths in overwrite mode and closes its own files. Caller-provided objects are
flushed but not closed; keep them open until the job finishes. Writes run off
the event loop, allowing another task to stop a job during file I/O.
wait() returns LocalJobResult and also sets job.result. The result
contains args (image followed by arguments), returncode, stdout,
stderr, and status. to_completed_process() adapts it to
subprocess.CompletedProcess when an actual exit status is available.
Exit 0:
COMPLETEDresult.Nonzero exit, including 125 or 137:
FAILEDresult with the actual code; no exception unless the caller invokescheck_returncode().Opening the stdout destination, image preparation, container creation, attachment, or startup failure:
LocalJobLaunchError.Output transport, exit-status retrieval, decoding, or file-write failure:
LocalJobExecutionError.Explicit stop before completion:
LocalJobCancelledErrorfromwait().Execution deadline or
wait(timeout=...):LocalJobTimeoutErrorafter cleanup is confirmed.Unconfirmed removal:
LocalCleanupErrororLocalCleanupTimeoutError. Cleanup failures take precedence over the execution error; inspect the exception’sresultwhen present, retain the handle, and retrystop().
Exceptions expose any available partial result. Cancellation or launch failure
may have returncode=None; the SDK does not invent an exit code. Decode or
sink errors may prevent constructing a result. stop() succeeds after
container creation and startup have settled, any owned container has been
removed, and output handling has finished. Image preparation may still be
finishing in the background; it cannot create a container after cancellation.
The execution timer starts when the Docker start request completes; image
pulling, container creation, and attachment are not included.
wait(timeout=...) bounds the entire wait, requests cancellation, and then
allows up to cleanup_timeout (default 30 seconds) for cleanup. A hung
daemon can prevent confirming removal: the SDK reports a cleanup failure instead
of claiming the container stopped.
Stopping during image preparation prevents container creation. An in-flight
Docker call may continue until it returns or produces another pull progress
message, at which point preparation releases its resources. Docker may retain
downloaded image layers for future jobs.
If a create response is lost, an absent container name does not prove that
creation will not finish later. Cleanup remains unconfirmed until the owned
container can be found and removed; retain the handle for retry or reconcile
the UUID label with the daemon after connectivity is restored.
Compatibility and installation
Local execution is an optional addition to v1. Install the [local] extra and
import from compute_horde_sdk.v1.local. Remote-only installations do not need
the Docker Python package or a Docker daemon.
Local jobs have their own result, status, and exception types; they are not interchangeable with remote jobs.
Validation
Disposable real Docker tests are opt-in:
CH_SDK_DOCKER_TESTS=1 uv run --extra local --group test pytest tests/integration/test_local_docker.py