ovstorage-user-choose-a-backend

Use when selecting the right ovstorage backend plugin for a task or deciding which route to configure.

SKILL.md

GitHub

Choose a Backend

Goal

Pick a backend kind whose address shape, capabilities, and credential model match the storage task.

Available backends

Backend Use when Address shape Notes
file Data is on a local or mounted filesystem visible to this process file:///absolute/path Best for local smoke tests, dev fixtures, and direct POSIX semantics.
http Data is read from an HTTP(S) endpoint https://host/path Read-oriented and URL-shaped; confirm write/list support before depending on it.
omniverse-storage-service Targeting a deployed Omniverse Storage Service gRPC; configured via address Needs address — either a discovery root (host of /api/v1/services and /api/v1/auth-config) with OIDC credentials, or a grpc:// / grpcs:// storage endpoint dialed directly. See per-backend section below.
s3 AWS S3 or S3-compatible object stores (MinIO, R2, B2, custom endpoint) s3://<bucket>/<key> aws_access_key_id + aws_secret_access_key (+ optional aws_session_token); env vars and ~/.aws/credentials honoured.
gcs Google Cloud Storage gs://<bucket>/<object> service_account_key (JSON blob) or ADC file_path; accepts service_account, authorized_user, external_account (WIF), and impersonated_service_account JSON; no IMDS.
azure Azure Blob and ADLS Gen2 (HNS-aware accounts) azure://<account>/<container>/<path> One of account_key, sas_token, service principal, or workload identity.
huggingface Hugging Face Hub repositories, read-only (models, datasets, spaces) hf://{models|datasets|spaces}/<namespace>/<name>/<path> (optional ?revision=<ref> pin); under an alias layer also the Hub's own https://huggingface.co/<namespace>/<name>/<path> URL Single token (descriptor default ${HF_TOKEN}); anonymous serves public repositories.
opendal Long-tail backends fronted by OpenDAL (services compiled in: fs, s3, webdav) opendal://<service>/<path> (configurable prefix) Driver-specific; s3 uses bare access_key_id / secret_access_key, webdav uses password.
nucleus NVIDIA Omniverse Nucleus content-collaboration server omniverse://server[:port]/path (with optional ?<branch>&<checkpoint> selector) Three credential methods (sso, userpass, api_token); refresh token persisted in OS keyring for warm-continue.
broker Routing traffic through an upstream ovstorage-broker daemon (per-call authz, broker-side caching, credential isolation) No scheme of its own — address roots are published by the upstream broker (typically s3://, gs://, azure://, omniverse://, file://) Single config key address (HTTPS discovery, direct gRPC, UDS, or named pipe). One credential method: client_credentials (OIDC client ID + secret) for headless deployments; interactive PKCE / device flow drives via Factory::authenticate.

Backend-specific notes

omniverse-storage-service

A Omniverse Storage Service deployment, accessed over gRPC with OIDC bearer-token auth. Configure with:

  • address — where the service is. Either an HTTP root that serves /api/v1/services and /api/v1/auth-config, from which the plugin discovers the gRPC endpoint and the OIDC client; or a grpc:// / grpcs:// storage endpoint, dialed directly with no discovery service and no OIDC.
  • oidc_client_name (optional, default "default") — picks an entry from auth-config.clients.

Credentials arrive via OAuth (PKCE for browser, device flow for headless). See ovstorage-user-authenticate-to-backend for the auth flow.

Pick this backend when you have an existing Omniverse Storage Service deployment; otherwise pick a cloud or file backend.

This backend connects to an existing service deployment. Configure its endpoint and credentials using the client plugin guide linked below.

s3

AWS S3 and S3-compatible object stores. Addresses are s3://<bucket>/<key>; the configured bucket is authoritative, so the prefix lives in the address, not the connection config. Config keys:

  • bucket (required) and region (required).
  • endpoint — optional; required for minio, b2, and custom compatibility profiles.
  • compatibility_profile — enum (aws, minio, r2, b2, custom; defaults to aws, or custom when endpoint is set). Selects addressing style and signing region per the matrix in plugin-s3.md; this is the knob to flip for R2 (auto signing region) or B2 (path-style + auto).
  • force_path_style (bool), force_request_payer (bool), profile (named entry in the AWS shared credentials file).
  • sqs_queue_url / sqs_max_messages / sqs_wait_seconds / sqs_visibility_timeout for watch_directory.

Credentials (aws_access_key_id, aws_secret_access_key, aws_session_token) are all optional; absent fields fall through the chain (env vars, then the AWS shared credentials file). IMDS, AWS SSO, and STS role assumption are not implemented — operators materialise a static-key profile out-of-band.

gcs

Google Cloud Storage. Addresses are gs://<bucket>/<object>. Config keys:

  • bucket (required).
  • project_id — optional; surfaced for operator clarity only.
  • service_account — optional; named entry in the credential chain.
  • endpoint — optional; for GCS-compatible deployments (e.g. fake-gcs-server).
  • pubsub_subscription / pubsub_pull_max for watch_directory.

Credentials are service_account_key (an ADC JSON blob of type: service_account, authorized_user, external_account, or impersonated_service_account) or file_path (ADC JSON file on disk, same four types; defaults to ~/.config/gcloud/application_default_credentials.json). Only service_account creds can mint V4 signed URLs; reads under every other credential type pull bytes through the JSON download endpoint instead. The GCE/GKE metadata server, the AWS / executable external_account source flavors, and workforce (user-identity) federation are not implemented in-process.

azure

Azure Blob and Azure Data Lake Storage Gen2 (HNS-aware). Addresses are azure://<account>/<container>/<path>. The Azure plugin rejects unknown config keys (CONFIG_KEYS is a strict whitelist), so don't add a prefix field. Config keys:

  • account (required) and container (required).
  • endpoint_suffix — optional; default core.windows.net (override for sovereign clouds).
  • blob_endpoint — optional full blob-tier URL (scheme, host, port, path prefix); overrides endpoint_suffix for that tier. Use https:// for real custom, private-link and sovereign endpoints. http://127.0.0.1:10000/devstoreaccount1 is the Azurite emulator form and is emulator-only.
  • dfs_endpoint — the same for the ADLS Gen2 dfs tier. May be set on its own (the blob tier then stays on endpoint_suffix); an HNS connection that sets blob_endpoint must set this too.
  • Prefer https://. A plain-http:// endpoint is accepted with any credential mode and on any host — http://azurite:10000 is a supported emulator shape, and needs no extra key. On a non-loopback host the connection logs a warn, because object bytes and metadata cross the link in the clear, a caller sas_token is readable in the request URL, an OAuth token in the Authorization header, and the redirect-following read and write paths hand out a five-minute Service SAS that can travel in the clear. Use http:// only for an emulator; for real custom, private-link and sovereign endpoints use https://.
  • hierarchical_namespace — bool; set to true for ADLS Gen2 HNS accounts. instantiate checks this matches the account's actual mode.
  • change_feed_enabled / change_feed_segment_lag_seconds / change_feed_poll_interval_seconds for watch_directory (flat Blob accounts only).

Credentials are picked from four distinct methods, in this resolution order: account_key (Shared Key signing) → sas_token (appended verbatim) → workload identity (federated_token_file + client_id + tenant_id) → service principal (client_id + client_secret + tenant_id). Managed Identity / IMDS, Azure CLI session reuse, and the rest of the DefaultAzureCredential chain are not implemented in-process.

huggingface

Hugging Face Hub repositories, read-only. One connection serves one repository; addresses are hf://{models|datasets|spaces}/<namespace>/<name>/<path>, with an optional ?revision=<ref> query pinning any address to a git revision. When an alias layer sits above the backend, the Hub's own repository URL resolves as well — https:// or http://, on huggingface.co, www.huggingface.co or hf.co, prefixed datasets/ or spaces/ for those repository types — with ?revision= carrying through; the Hub's /blob/, /resolve/ and /tree/ web-view links are not unwrapped. Config keys:

  • repo_id (required; namespace/name).
  • repo_type — optional; model (default), dataset, or space.
  • revision — optional; the connection's default revision (default main).

The single credential is token (a hf_… access token; read scope is sufficient), with a descriptor default of ${HF_TOKEN}; an empty bundle connects anonymously and serves public repositories. Writes, deletes, metadata updates, and directory watches all answer Unsupported; version listing walks the repository's commit history.

opendal

Long-tail backends fronted by Apache OpenDAL. Pick a service — the descriptor's enum advertises only the services this workspace compiles in, namely fs, s3, and webdav. Other OpenDAL services (http, sftp, hdfs, etc.) are not enabled and will be rejected at instantiate. Config keys:

  • service (required enum: fs | s3 | webdav).
  • endpoint — optional; passed through to the chosen driver.
  • config_json — optional flat JSON string-map of driver knobs (bucket, region, root, username, ...).
  • prefix — optional URL; route prefix shown to callers (defaults to opendal://<service>/).

Credentials are a fixed shared set: access_key_id + secret_access_key (for the OpenDAL s3 driver), password (for WebDAV), and private_key. Note that the OpenDAL s3 driver uses the bare access_key_id / secret_access_key names — not the aws_*-prefixed names that the first-party s3 plugin uses — so the credential shapes are not interchangeable. For S3-shaped storage that needs the full conditional-write / version-listing capability set, prefer the first-party s3 plugin.

nucleus

NVIDIA Omniverse Nucleus is the content-collaboration server for the Omniverse platform. Pick this backend when:

  • You have an Omniverse Nucleus server deployed (omniverse:// URLs).
  • You need checkpoint-based versioning (Nucleus tracks per-path checkpoint history; list_versions enumerates them newest-first and an ?&<checkpoint> URL selector pins reads to a specific checkpoint).
  • You need ACL-aware permissions on stat (populates_effective_permissions_on_stat = true).

Configuration is minimal: server (the host[:port] of the Nucleus server) is the only required key. endpoint overrides SOWS discovery for production session establishment; prefix scopes which paths the backend instance serves (default /); use_lft (default true) gates the LFT redirect path for bulk bytes.

if_match on this backend is an opaque etag string — Nucleus's mutating omni1 method that carries a conditional (update_asset) accepts only an etag. Pass if_match: Some(etag_string). The checkpoint selector lives in the URL (?&<N>), not in if_match.

See ovstorage-user-authenticate-to-backend for the three credential methods (sso, userpass, api_token) and the keyring-backed refresh-token warm-continue.

broker

Routes traffic through an upstream ovstorage-broker daemon. The broker holds long-lived cloud credentials, enforces per-call authorization, runs broker-side metadata + byte caching, and keeps credentials off the calling host. Pick this backend when:

  • You want a credential boundary: the calling process holds only short-lived broker bearers, not S3 / GCS / Azure keys.
  • You want per-call authorization (allow / deny / audit) against a configured built-in authorization policy.
  • You're running zero-trust fleets, workload-identity-federated deployments, or want a shared metadata cache in front of the cloud.

Configuration is minimal: address (broker discovery URL or direct gRPC endpoint) is the only required key. Accepted shapes:

  • https://broker.example.com — HTTPS discovery (fetches /api/v1/services for the broker endpoint).
  • http://localhost — plaintext-HTTP discovery (honored only for localhost, IP literals, and .local hosts).
  • grpc+tls://host:port / grpc+tcp://host:port — direct gRPC, skipping discovery.
  • /path/to/socket — Unix domain socket.
  • pipe:NAME — Windows named pipe.

Optional oidc_client_name (default "default") selects an entry from the broker's published auth-config.clients.<name> when the deployment publishes auth-config under a non-default key.

Address roots come from the broker on ListAddressRoots after instantiate. The library merges them into the routing table; the per-route capability profile mirrors the upstream plugin's profile field-by-field (so a brokered s3://corp-prod/ route advertises the same supports_recursive_list, supports_version_listing, version_list_order, watch_directory_kinds, redirect_size_threshold, etc. it would in Direct mode).

See ovstorage-user-authenticate-to-backend for the client_credentials credential method and the broker-driven interactive PKCE / device flow.

Recipe

  1. Run ovstorage-user-getting-started and confirm the backend kind appears in backend_kinds. That list is the layers this stack declares, so a kind you know the library supports — file included — is absent until the config declares a layer for it.
  2. Match the address root you need to the backend's URL scheme and configured address_roots. A public s3://, gs:// or azure:// address may be readable with no route of its own when the host grants an anonymous fallback for that scheme — read-only, and anonymous in the strict sense (no credential of any kind is attached); the sign is a hidden s3:/// root in the doctor report. Every mutation through it answers Unsupported, so configure a connection when you need to write.
  3. Check capabilities before relying on writes, recursive list, directory delete, watches, or optimistic concurrency.
  4. Follow ovstorage-user-authenticate-to-backend before first use if the descriptor has credential fields.

References