ADR 0001: Elixir owns the API, Rust stays optional

Copy Markdown

Status

Accepted. Stage 1 decoded the footer in Elixir. The eight committed fixtures match the upstream reader's metadata JSON, so the Rust FlatBuffers bridge was not added. Stage 2 decodes one CASCADE EncUnit in Rust and returns owned Elixir terms. Arrow buffers do not cross the NIF.

Context

F3 files end with a 32-byte postscript that is ordinary little-endian fields plus the magic F3. The footer and column metadata are FlatBuffers. The payload is Vortex (CASCADE) or an embedded Wasm decoder (CUSTOM_WASM). Upstream fff-poc is a nightly-Rust research crate with a process-global mimalloc allocator, and its Vortex and Lance dependencies are git forks.

Elixir callers need {:ok, value} and {:error, %F3.Error{}}, explicit close, and a library that loads without a native toolchain. The postscript has to stay readable when the NIF is absent.

Options

  1. Call fff-poc directly from Rustler. Rejected. The crate needs nightly Rust and a global allocator, and its public writer and reader types are experimental.
  2. Decode FlatBuffers only in Rust, using the flatbuffers 24.3.25 crate and the generated schema. Possible later, as a small crate that does not link Vortex. It is not required to open a file or to read the postscript.
  3. Decode the postscript, and then the footer tables Stage 1 needs, in Elixir. The postscript is fixed-width. FlatBuffers is a documented binary layout, and the Elixir walker can be checked against the pinned schema and the upstream reader. A Rust FlatBuffers bridge remains the fallback if those differential tests fail.

Decision

Elixir owns the public API, errors, source lifecycle, version policy, the postscript, and the footer. Stage 1 decodes the FlatBuffers footer with a bounded pure-Elixir walker checked against pinned File.fbs and the upstream reader. A small stable-Rust flatbuffers 24.3.25 bridge is the fallback only if those tests fail. It was not needed.

Stage 2 adds decode_encunit/3 behind F3_BUILD_NATIVE=1. The crate depends on Vortex at c0d89a66cbaa542bd9aa277cfdf52f2728c4b37a and arrow-array 53.3.0. It does not depend on fff-poc and it does not install mimalloc. The function is scheduled DirtyCpu. Stable Rust cannot compile Vortex, and nightly-2025-02-24 cannot compile Rustler 0.38, so this crate pins nightly-2025-11-15. See ../upstream-compatibility.md.

Do not compile fff-poc into f3_native. Stage 3 compiles Wasmtime 28.0.0 into the same crate. It does not depend on fff-ude-wasm. The host runs only after Elixir has selected :wasm.

Stage 4 writes through an out-of-process helper, f3_write, compiled as a bin of pinned fff-poc on nightly-2025-02-24. The helper's process has the mimalloc global allocator. The BEAM does not. f3_native still does not link fff-poc.

Consequences

  • mix test runs without Rust.
  • Postscript inspection cannot be broken by a NIF load failure.
  • Stage 1 walks vtables in Elixir with a depth cap and a visit cap. The fallback remains a small Rust parser, not a fork of fff-poc. It was not needed for the committed fixtures.
  • Stage 2 decodes CASCADE units through that narrow NIF. Elixir projects, stitches nested columns, gathers a local dictionary, and slices batches.
  • Wasm is opt-in. wasm: :off refuses CUSTOM_WASM without reading the section. wasm: :sandbox calls decode_general_ffi under fuel, memory, and module-size caps. Elixir extensions are registry policy, not a file field.

Revisit when

  • Stage 1 cannot match the upstream reader's metadata on the committed fixtures.
  • A NIF must keep Arrow buffers alive, or cancellation has to cross the scheduler.
  • Upstream publishes a stable crate that builds on stable Rust without a global allocator.