Architecture

Copy Markdown

The library owns the byte-source lifecycle, the postscript, the footer, and the public read API. Rust decodes one CASCADE encoding unit at a time and returns owned Elixir terms. Arrow buffers do not cross the NIF. Wasm runs only when a read passes wasm: :sandbox.

flowchart LR
  Read[F3.read or F3.stream]
  Meta[F3.Footer]
  Slice[pread one EncUnit]
  Nif[dirty CPU NIF]
  Batch[F3.Batch]
  Read --> Meta --> Slice --> Nif --> Batch

Components

F3 is the public module. open/2 builds an F3.File around F3.IO.Local, F3.IO.Memory, or F3.IO.HTTP. The HTTP source is opt-in. It sends one HEAD, then one ranged GET per pread/3, and it does not follow redirects. close/1 returns the updated handle and is idempotent on that handle. schema/1, metadata/1, and inspect/2 decode the tail when the handle is open. read/2 returns columnar batches. stream/2 returns a stream and does not pread until the first pull.

F3.Format holds the pinned magic, the 32-byte postscript layout, and compatible?/2. The only accepted wire version is 0.1. F3.Postscript reads that layout, rejects a short tail, a bad magic, any other version, a compressed footer, and a claimed size past the file or past max_metadata_bytes:. The default cap is 16 MiB and is an argument, not a process global.

F3.FlatBuffer walks one buffer: root offset, vtable, scalars, strings, vectors, and unions. Offsets are checked against the buffer length. Depth and the number of visited tables are capped. F3.Footer turns the footer and the out-of-band column-metadata roots into F3.Schema, F3.Metadata, and F3.RowGroup. Arrow types the fixtures do not use become {:unsupported, id} and do not fail the file. Wasm section offsets are recorded and the bytes are not read. Checksums are recorded and not verified.

F3.Batch stores columns keyed by logical field name. to_rows/1 zips them into maps and is the convenience path. F3.Writer checks arguments in Elixir, then streams batches to an out-of-process f3_write helper. The helper is fff-poc's FileWriter on nightly-2025-02-24. close/1 renames a temp file in the destination directory onto the path. That rename is local-filesystem behavior. Without F3_BUILD_NATIVE=1, a valid open/2 returns :not_implemented and creates nothing.

F3.Reader selects columns and row groups, checks a chunk checksum when the footer stored one, and slices a decoded unit to batch_size (default 8192). F3.Logical stitches unnamed physical columns into the logical tree. F3.Native is compiled only when F3_BUILD_NATIVE=1. decode_encunit/3 and decode_wasm/5 run on the dirty CPU scheduler. The crate calls Vortex, Arrow, and Wasmtime 28. It does not link fff-poc and it does not install a global allocator. Without the NIF, a CASCADE read returns :native_unavailable. A CUSTOM_WASM read with wasm: :off returns :unsupported_encoding without the NIF. Metadata inspection still does not execute a module.

There is no Application callback, no supervision tree, and no process registry. See decisions/0001-hybrid-elixir-rust.md.

What stays off

Wasm stays opt-in. A shared dictionary is still :unsupported_encoding. A local dictionary is gathered in Elixir after the NIF returns the values and the indexes. lib_urls stay unread. An HTTP source does not fetch them and does not follow redirects. The writer does not embed Wasm and does not write to a remote URL.

Fixture path

scripts/fixtures/generate.sh clones the pinned commit into gitignored tmp/upstream-f3, copies scripts/fixtures/generator/src/main.rs to fff-poc/src/bin/f3_fixture_gen.rs in that checkout, and builds it with cargo build --locked. scripts/fixtures/dump_metadata.sh runs the pinned reader once and writes test/fixtures/golden/*.json. mix test checks the manifest and those JSON files. It does not compile upstream.