smolvault

smolvault is an immutable, content-addressed vault for everything you have — that speaks just enough HTTP to be mistaken for a local disk.

One file, zero dependencies, no install step. Drop files into the terminal to seal them write-once (WORM), deduplicated by content and verified on every read — then stream any of it to mpv, your phone, or a browser with instant seeking. Media is the headline act; documents, archives and disk images are equal citizens.

git clone https://github.com/smolfiddle/smolvault.git
cd smolvault
python3 smolvault.py        # that's the whole install

• Backups you can't corrupt by accident. Files are sealed forever: overwrite → 409, delete → 403. Bit-rot fails loudly instead of silently.
• Storage that shrinks. Content-defined chunking + dedup means identical content costs zero extra bytes, across files and folders.
• Remote without setup. One command serves the vault on your LAN; phones, TVs and other machines get full read/write parity — discovery included, IPs optional.
• It gets out of the way. Single-file stdlib Python: copy it anywhere, run it, done.

python3 smolvault.py                    # creates vault.vault + opens the hub

Press a, drag any files from your file manager into the terminal, Enter:

  ✓ sealed /movie.mkv  8.4 GB · 9 chunks
  ── 1 added · 0 skipped · 0 failed · 8.4 GB of 8.4 GB newly stored (0% saved)

Point it at a folder to ingest a whole tree — or [b] to open the browse navigator: a tiny file manager that takes over the screen, starting at your current directory and locked to it (you can descend into subdirectories, never climb out). ↑↓ move · → descend · ← up · Space toggles (on a folder = its whole subtree, ◐ n/m while partial) · :a select-all · :u clear · :s seal — every other printable key types into the filter. Vault internals (*.vault, __pycache__) never appear. Then press p and start typing — results filter as you type:

  watch ❯ dune · 2
   ❯ /movies/dune.2021.mkv      (8.4 GB)
     /movies/dune.part.two.mkv  (9.1 GB)

Enter plays in mpv. When an episode ends, [Enter]=next chains the season in real numeric order (E2 → E10). The banner shows a LAN URL — open it on your phone and the same library streams there.

  ┌────────────────────────────────────────────────────────┐
    smolvault 0.4.1
     vault    vault.vault  · 12 files · 22.3 GB logical · 11 GB stored
    local    http://127.0.0.1:8100/
    network  http://192.168.1.14:8100/   ● running   ← phone/TV ready
    auth     password protected · AES-256-GCM at rest
  └────────────────────────────────────────────────────────┘

    a add       drop files · path · [b]rowse a folder
    s search    find something
    p play      search → mpv (w) · [b]rowse vault
    l library   browse everything · [b]vault browser
    d du        space by folder · find double-sealed files
    m board     this vault's messages · clients & server post here
    i info      details about a file · [b]browse
    g get       export a copy · [b]browse vault
    c copy      stream link → clipboard · [b]browse vault
    y sync      mirror another smolvault
    S server    start/stop network sharing
    P port      change port
    v verify    check every chunk's hash
    q quit

After each action the hub collapses to a one-line prompt — ❯ vault.vault · 12f ● :8100 ❯ — so long sessions stay calm. The menu is one h away.

• WORM immutability — sealed means sealed; perfect for media libraries, evidence, datasets, anything that must not quietly change. Race-tested: 12 parallel same-path PUTs yield exactly one seal.
• Streaming-grade HTTP — HTTP/1.1 keep-alive, full RFC 7233 byte ranges (N-M, N-, -N), If-Range, 416s, ETag = content hash, Cache-Control: immutable. Players seek as if the file were local.
• Content-defined chunking — gear-hash CDC (64 KB–1 MB chunks) makes dedup work across shifted/ressembled files, not just identical copies: a mid-file edit still dedups roughly half or more of the file on re-seal (up to ~98%).
• Entropy-adaptive storage — high-entropy media stored raw and scanned on a coarse fast stride (~2× seal speed); compressible text is compressed.
• Folder-aware ingest — recursive walks keep structure (s01/e1.mkv → /movies/s01/e1.mkv), with a live Space-toggle multi-picker.
• Vault sync — additive gap-filling replication between instances; nothing ever deleted, safe on a schedule.
• Remote ingest (opt-in) — run the server with --share-root DIR and password-holding clients can browse that directory remotely and seal server-local files into the vault (traversal-locked, additive).
• Space insight — --du breaks storage down by folder and surfaces byte-identical files sealed under different paths (WORM keeps them all, so knowing matters).
• Integrity on every read — per-chunk BLAKE2b-256 verified end-to-end; --check scrubs the whole vault. Crash-tested: kill -9 mid-ingest recovers clean (--gc + --check).
• At-rest encryption (opt-in) — every chunk sealed with AES-256-GCM via the system's OpenSSL; key derived from your vault password (scrypt). Zero PyPI dependencies. Dedup fully preserved.
• Message board, live — each vault carries a small public board (MMO-style): clients and the server post notes and system events (sealed 12 files, sync pushed X). Press m: new messages stream in as they arrive while you type.
• Optional password — PBKDF2-HMAC-SHA256 over HTTP Basic; doubles as the encryption key wrapper, and the network gate it controls is an independent switch (--auth on|off) so trusted LANs can stream freely.
• Port picker at vault creation — port [8100] (Enter=auto next free) via prompt_for_port, persisted per-vault; change anytime with P in wizard or --port / $SMOLVAULT_PORT.
• Tiny vault file manager — [b]rowse vault in play/get/library/info/copy (and remote watch/get) — same raw-mode keys (↑↓ → ← Enter i info), live filter, backed by VaultBrowseState (prefix tree from the listing, with the row carried through so the renderer stays O(1) per visible row). get and library open it in multi-select: Space to check, :s to act on everything checked.

python3 smolvault.py                    # hub, as shown above
# on first vault creation you'll be asked: port [8100] (Enter=auto next free)
# or set upfront: python3 smolvault.py my.vault --port 9000
# or env: SMOLVAULT_PORT=9000 python3 smolvault.py

python3 smolvault.py lib.vault --add /data/movies          # whole folder, structure kept
python3 smolvault.py lib.vault --add a.mkv notes.txt --into /docs/
python3 smolvault.py lib.vault --list
python3 smolvault.py lib.vault --du
python3 smolvault.py lib.vault --search marvel
python3 smolvault.py lib.vault --play dune [--player mpv]
python3 smolvault.py lib.vault --info dune
python3 smolvault.py lib.vault --get dune -o copy.mkv
python3 smolvault.py lib.vault --check                     # scrub all chunks
python3 smolvault.py lib.vault --gc                        # prune orphaned chunks
python3 smolvault.py lib.vault --encrypt                   # at-rest AES-256-GCM (resumable)
python3 smolvault.py lib.vault --decrypt                   # strip encryption

--du output at a glance:

  space by folder:
  /movies        12 files ·     14.2 GB
  /docs         210 files ·      1.8 GB
  ── vault: 16.0 GB logical · 10.3 GB stored · 36% deduped

  byte-identical sets (WORM keeps them all):
    8.4 GB ×2  /movies/cut.mkv
               = /movies/old/cut.mkv
  ── 1 set · 8.4 GB sealed twice under different names

Exit codes: 0 ok · 1 no match/refused · 2 usage error — pipe-friendly.

The same file is client and server. Copy it anywhere:

python3 smolvault.py lib.vault --serve                  # plain server + live feed
python3 smolvault.py --connect                          # discover vaults on the LAN
python3 smolvault.py --connect 192.168.1.14:8100        # or aim directly
SMOLVAULT_SERVER=host:port python3 smolvault.py --connect

Full parity over the network: list, live search, watch in your local player (streams go straight from the vault — never proxied), export, even remote drag-paste uploads. Discovery runs over UDP broadcast (--no-discover silences it).

Because vaults are immutable and content-addressed, syncing is pure gap-filling — zero conflict logic:

python3 smolvault.py lib.vault --sync-to   192.168.1.20:8100   # push mine → theirs
python3 smolvault.py lib.vault --sync-from 192.168.1.20:8100   # pull theirs → mine

Additive-only: fills gaps, skips already-sealed paths, deletes nothing. Every pulled file is chunk-hash-verified on arrival. A transfer plan prints first; in the wizard, y sync adds a LAN discovery picker.

No config files — flags and environment only. A few things are remembered per vault in its own SQLite config table (the last LAN port, whether the network gate is on, and an exposed --share-root); there is no separate config file to edit.

A locked vault answers 423 to every endpoint, /__api/* included — no metadata leaks through the listing endpoints.

Framing is strict: a request carrying Transfer-Encoding, or two conflicting Content-Length headers, is rejected before any handler runs, and every error response either drains its request body or closes the connection. Ambiguous framing is how HTTP request smuggling starts.

Auth (if set): HTTP Basic, PBKDF2-HMAC-SHA256, 100k iterations. Disable it for trusted LANs with --auth off — encryption stays on.

Loopback · NVMe · 6 cores · Python 3.12 (full sheet, regenerate with python3 benchmark.py, ~7 min). Every figure below comes from that one committed run:

The server itself runs inside a hard 1 GB RAM / no-swap / 4-core cgroup envelope (systemd-run --user -p MemoryMax=1G) while a 700 MB movie is sealed and streamed back out of it:

Note: PRAGMA mmap_size=512M + cache_size=-64000 are tunable. The page-cache figure is a cap, not a reservation — SQLite grows it lazily, so a connection only holds the pages it actually touches. Lowering it to 2 MB was measured and reverted: with a chunks index past the cap (~300k chunks), range reads degrade p50 1.6 → 4.9 ms and p95 7.1 → 12.7 ms (fresh process per setting, identical warm-up). Seeks are the whole point of a media server, so leave it alone unless you have measured a problem. MemoryMax=1G leaves headroom; a real Pi will ingest slower, while the memory profile holds.

Memory stays flat because files stream in chunks — the ceiling is your disk, not your RAM. Reproduce with python3 benchmark.py --lowmem (Linux + user systemd session). Caveat: this is x86 under a memory ceiling, not ARM silicon — a real Pi will ingest slower, but the memory profile holds.

Every promise the docs make, measured, from the same committed full sheet:

The WORM race test earned its keep: it exposed a real bug (losing writers stalled 60 s on SQLite lock timeouts, then died without a response) which is now fixed with explicit single-writer discipline — contenders get instant 409s instead.

Reading these numbers. Several sections seal os.urandom data, so chunk counts and throughput move run to run — the CDC-resync dedup figure alone has ranged 49–95% across runs purely on where the edit landed, and the 700 MB seal has landed anywhere from 34 to 49 MB/s. On this hardware the media-seal path has a pass-to-pass spread exceeding 2×, so treat single-run deltas under ~20% as noise. The numbers above are one full run, not a curated best-of; if a figure here looks unflattering, it unflattered the run that is committed.

"Could not connect to socket" (Termux intents) — your Termux app build predates the AM socket server (needed on Android 12+/14+ where raw am is blocked). Fix: update the Termux app itself from github.com/termux/termux-app/releases (≥ 0.119), fully close & reopen, then pkg reinstall termux-am. smolvault also auto-falls back to termux-open, which bypasses am entirely.

LAN devices can't reach the vault

1. Firewall: sudo ufw allow 8100/tcp && sudo ufw allow 8100/udp
2. Router AP/Wireless Client Isolation enabled (common on ISP routers) — disable it, or use a hotspot: nmcli dev wifi hotspot ssid vaultnet password …

Port already in use — smolvault remembers the last port per vault and suggests the next free one.

What encryption protects: a stolen disk, SD card, laptop or leaked backup of vault.vault. Chunks are unreadable without your password.

What it does not protect:

• the running server (keys live in RAM while unlocked),
• network traffic — smolvault speaks plain HTTP; put it behind Tailscale, WireGuard or an SSH tunnel for remote access,
• content equality: nonces derive from chunk hashes so dedup keeps working, which means an attacker who already knows a file's exact bytes can confirm it exists in your vault.

Auth vs encryption are independent: encryption always protects the stored bytes; whether network requests need the password is a separate switch (--auth on|off, default on). Media-server on a trusted LAN? --auth off gives players password-free streaming while everything stays sealed on disk.

Passphrase strength matters: the key is only as strong as the password wrapping it (scrypt, n=2¹⁵). Password changes re-wrap the master key in milliseconds — data is never re-encrypted. --decrypt refuses to drop key material while any chunk is still sealed, so a half-finished migration can never look like successful one. The message board is visible to anyone holding the vault password and is not replicated by sync. --share-root grants password holders read+seal access to that one directory (traversal- and symlink-locked, additive-only, persisted in vault config) — point it at a downloads folder, never at / (/ is now refused). Paths are canonicalized (/a/../b → /b), truncated uploads are rejected (400) and rolled back, and gc holds a write lock plus an ingest lease to avoid orphans.

smolvault is the distilled successor of DenseVault. Deliberately absent: delta encoding, WebDAV lock theater, hidden system collections, compression pipelines, config files, worker-thread ingest pipelines (measured slower on boost-heavy consumer CPUs — the GIL-bound chunker runs fastest alone). Deliberately kept verbatim: the gear-hash chunker and hash-verified reads; the entropy gate now also picks each file's sealing stride.

One invariant worth knowing if you ever touch the ingest path. The chunker scans only as far as its buffer reaches, so cut positions depend on read sizes as well as content. Ingest always feeds it a 256 KiB entropy probe followed by the reader's own 4 MiB reads, which is what makes chunking deterministic and dedup/resync work. Swapping in a buffered or whole-file reader would silently move every boundary — and --check would still pass, because each chunk stays internally consistent. If you change how _put builds its reader, re-measure the CDC-resync dedup row in BENCHMARKS.md before trusting it.

No single pillar here is new — CDC chunking comes from the backup world (borg/restic/FastCDC), WORM-over-HTTP from stores like verm and S3 Object Lock, and stdlib-Python range-streaming servers are practically a genre. What we could not find elsewhere is all of it at once in one dependency-free file: write-once + content-defined dedup + verified reads + RFC-7233 streaming + LAN discovery + client/server parity. The CAS engines (casq, Kloset, farchive) stop before the serving layer — several advertise "no network" as a feature; the tiny Python servers (servery, neev, pi-media-server) serve plain filesystems with none of the storage semantics. smolvault lives in the gap between those two camps.

Issues and PRs welcome — it's one file on purpose; keep additions honest about that budget. Reproduce before/after with python3 benchmark.py.

MIT — see also the header of smolvault.py.