ccr — reversible payload compression (the payload axis)

ccr compress <file|-> · ccr retrieve <handle> · ccr stats

Shrink any blob — a tool output, a log, a RAG chunk, a file — before it enters context, while keeping the original recoverable. The payload-axis counterpart to a knowledge-graph loader (the retrieval axis): kg shrinks which entities reach the model; ccr shrinks each blob that does.

Lifted from the CCR / reversible-compression component of Headroom (28.5k★, Apache-2.0). We lifted the pattern, not the dependency. Anchor: BRO-1521.

The one idea

Compression is reversible because the full original is cached locally, keyed by content hash. The model sees a compact lossy view + a handle (ccr://<sha256>); it calls retrieve(handle) only when it needs the bytes.

So the view can be aggressively lossy without losing information — the loss is recoverable on demand. Same shape a KG loader uses on the other axis: a lossy catalog projection that expands to the full entity body on load.

Mechanism

Dispatch by detected content type (a file-extension hint beats the content heuristic, which beats nothing):

Type	Compact view	Reversible?
json	type/shape skeleton (keys + value types + collection sizes; no leaf data)	yes (cache)
code	structural outline (imports + def/class/fn signatures w/ line numbers; bodies elided)	yes (cache)
text	head + tail by line, then head + tail by character on any over-long line	yes (cache)

Deterministic, stdlib-only, no ML — the value is the reversible-cache architecture, not the compression ratio. Identical payloads are content-addressed (idempotent: one handle, one record). The view is never emitted larger than the original (tiny inputs fall back to the payload).

Two budgets, and they apply on both line-oriented paths. --head/--tail bound the number of lines; --line-budget (default 2000 chars, 0 disables) bounds each line. Without the second, a payload that is one enormous line — a minified or truncated JSON blob, a single-line dump — compressed to exactly 0%, because len(lines) <= head + tail returned the whole thing as the "compact view". Truncated JSON lands here by construction: detect_type needs a successful json.loads, so a streamed/cut-off document routes to text.

The budgets are independent: --line-budget 0 (or any value ≤ 0) disables the character budget only, and --head/--tail keep eliding. It used to disable neither cleanly — for multi-line input len(ln) <= 0 was false for every non-empty line, so disabling the budget routed the payload down the join path it was meant to avoid: compact_text("a\r\nb\r\n", line_budget=0) returned "a\nb", dropping a trailing newline, normalising CRLF, and reporting a 50% saving with nothing marked elided. A "no compression" flag that silently rewrites bytes is worse than one that does nothing.

The char budget applies to the code outline too. It initially did not: compact_code(payload, **_) swallowed line_budget, so a minified JS/TS bundle — one enormous line that still matches _DEF_RE, and the canonical artifact this skill exists to shrink — emitted that whole line as its "outline" and still compressed to 0.0%, with no flag able to work around it. The same defect, surviving on the sibling path. Signature lines and the no-structure compact_text fallback now both carry the caller's budgets.

Guarantees

Each row names the mechanism that holds the invariant, and is backed by a mutation-proven test — revert the fix and the named test fails.

One scope note, stated rather than glossed. _is_confined's is_symlink() guard is defence-in-depth with no reachable behavioural difference: delete it and the suite stays green, because content-addressing means a symlink can only ever serve content whose sha256 matches the requested handle, and no content hashes to the link's own name. It is an equivalent mutant, not an untested guard, so no test is claimed for it.

That distinction is the point. An earlier revision of this file asserted "each is backed by a mutation-proven test" as a blanket claim; adversarial review showed it was false for the hex-validation row, whose outcome was upheld by a neighbouring guard while its stated ordering went unpinned. That gap is now closed by test_malformed_handle_is_refused_BEFORE_the_filesystem_is_touched, which distinguishes the two by pointing the cache at a nonexistent directory so the guards would give different errors. A blanket "everything is proven" line is itself an unverified claim — the exact failure mode this skill's review process exists to catch.

Invariant	What holds it
Byte-exact. retrieve returns the input's exact bytes: CRLF, lone CR, missing trailing newline, NUL, BOM, 4-byte UTF-8, lone surrogates, and raw non-UTF-8 bytes all survive.	compress reads bytes (read_bytes + surrogateescape) — read_text applies universal-newline translation, which rewrote \r\n → \n before hashing, destroying the original. retrieve writes through stdout.buffer.
Confined. A handle is [0-9a-f]{8,64} and nothing else, validated before it touches the filesystem; the resolved record must be a real file inside the cache dir.	Handles come back from model output — the untrusted side. Path.__truediv__ silently discards the cache dir for an absolute token and keeps .. for a relative one, and a <sha>.json symlink can point anywhere.
Atomic. A concurrent reader never observes a partial record.	Records are published by temp-file + os.replace, never streamed into the final name.
Verified. Every read recomputes the sha256 and refuses a mismatch — retrieve, compress's self-heal check, and stats.	A content-addressed store that hands back unverified bytes is not content-addressed. stats was the one read that opted out, so a record retrieve refused as tampered was still rolled into the totals.
Honest rollup. stats derives its totals from the digest-covered field (len(original)), and reports every excluded entry with the reason that excluded it (unusable_reasons), never a lump sum.	The sha256 covers original and nothing else, so the stored original_chars counter sits outside it: editing it to 1e9 kept the digest valid and reported cumulative_saved_pct: 100.0. compact_chars cannot be recomputed (the view is not stored), so it is accepted only inside the range every record satisfies by construction: 0 ≤ compact ≤ original. A lump unusable count invited the CLI to narrate a cause it never checked — "digest mismatch … re-compress to heal" is false for a stray hello.json, a dangling symlink, or a directory named adir.json.
Self-healing. A truncated or tampered record is treated as absent, so the next compress of that payload rewrites it.	The old if not rec_path.exists() short-circuit made a partial record permanent.
Self-collecting. A .ccr-tmp-*.part orphaned by SIGKILL or power loss is unlinked at the next cache open, once older than a 6h TTL.	_write_record's cleanup covers a failed publish; a hard kill runs no handler, and nothing else in the store looks at .part names, so the leak was permanent. Age, not an exclusivity check, gates the delete: mtime advances while a live publish writes, so an hours-stale temp file cannot be one. The window is measured — instrumenting os.replace across 640 concurrent publishes puts the longest a .part was ever live at 15ms (median 0.1ms), so a 6h TTL carries ~10⁶× margin on its own evidence. Held under concurrency by test_concurrent_writers_and_a_reaper_produce_no_corrupt_records (4 writers + a continuously-reaping 5th, in CI, ~0.15s); an ad-hoc 16-writer run of the same shape produced 640 records, 0 failures, 0 corrupt.
Private. Cache dir 0700, records 0600.	The cache holds full plaintext originals.
Linear. Both code regexes are line-bounded ([ \t]*, [^\n]*).	^\s* matches \n, so _IMPORT_RE ran to EOF and backtracked at every line start — quadratic (32KB of blank lines took 5.6s), reachable from --type auto.

--json is a machine envelope: always pure ASCII, so it stays pipeable even when the payload holds bytes no JSON reader could otherwise decode.

Addressing. For valid-UTF-8 input the handle is exactly sha256(<the file's bytes>). Input that is not valid UTF-8 is addressed by its surrogateescape-decoded form instead, so the handle is not sha256sum of the file — this keeps the digest collision-free (a surrogateescape digest maps a lone-surrogate string and a file holding the corresponding raw bytes onto the same handle). Byte-exact recovery holds either way.

Usage

python3 scripts/ccr.py compress path/to/big.json          # view on stdout, savings on stderr
cat huge.log | python3 scripts/ccr.py compress - --type text --head 30 --tail 10
python3 scripts/ccr.py compress minified.json --line-budget 4000   # per-line char budget
python3 scripts/ccr.py retrieve ccr://<sha256> > out.bin  # byte-exact original (full handle or unique prefix)
python3 scripts/ccr.py stats --json                       # cache size + cumulative savings

Exit codes: 0 ok · 1 no such / malformed handle, bad payload, user error · 2 internal error.

As a library:

import ccr
r = ccr.compress(payload, filename="server.ts")   # auto-detects code
context_blob = r["view"]                            # feed this to the model
original = ccr.retrieve(r["handle"])                # expand on demand

BROOMVA_CCR_HOME relocates the content-addressed cache (default ~/.cache/broomva/ccr/, created 0700) for CI runners and non-standard layouts.

Tests

python3 -m pytest skills/knowledge/ccr/tests/ -q     # 55 tests
python3 skills/knowledge/ccr/tests/test_ccr.py       # no pytest needed

Core invariant under test: the view is lossy, but retrieve(handle) returns the original byte-for-byte — asserted through the CLI over a byte-level fixture matrix (\r\n, lone \r, no trailing newline, NUL, BOM, 4-byte UTF-8, lone surrogate, raw non-UTF-8) as well as through the Python API. Plus: 64-char traversal/absolute/symlink handle rejection, cache atomicity + digest verification + self-healing, 0700/0600 permissions, ReDoS bounds on both code regexes, the character budget, head/tail clamping, and deep-JSON recursion.

The temp-file reaper is tested at the shape that produced it: a child process SIGKILLs itself from inside os.replace, and the test asserts the whole lifecycle — the orphan appears, no record lands at the canonical path, a young orphan survives (it is indistinguishable from another process's live publish), and only a stale one is collected. A second test races the reaper against a real in-flight publish rather than asserting the mtime rule statically: remove the age guard and the publish dies with FileNotFoundError. A third runs the whole thing concurrently — 4 writer processes plus a fifth reaping continuously — because a delete path in a shared directory is where a reaper's regressions actually live, and a claim about concurrency that only exists in a changelog is not re-verifiable. It asserts the reaper collected nothing: every .part in that directory belongs to a running writer.

Scope limit, stated rather than left implicit. Age does not cover a forward clock step larger than the TTL (VM snapshot restore, an RTC that NTP then corrects). Forcing that case — a reaper at ttl=0 against 640 concurrent publishes — makes publishes fail loudly with FileNotFoundError (6.2% on one run, 12.7% on an independent reviewer's; load-dependent) and corrupted 0 records in both. An availability fault on a retryable operation, not an integrity one.

Tests assert at the layer that can actually break. test_small_text_not_mangled asserts compact_text directly — routed through compress() it cannot fail, because the len(view) >= len(payload) clamp rescues even a deleted compactor. Likewise the traversal tests assert _resolve_sha directly, since the downstream digest check would otherwise mask an escape.

Install

npx skills add broomva/skills --skill ccr

Provenance

• Origin: /checkit github.com/chopratejas/headroom → entity tool/headroom
• Pattern sibling: the kg loader (retrieval axis) / llm-as-index-architecture
• Ticket: BRO-1521