Why another file format? Introducing F3

Copy Markdown

Status: technically reviewed
Audience: Elixir engineers who already know Parquet or another columnar format well enough to use it, and who have not read the F3 paper.
Estimated reading time: 8 minutes, estimated from length, not from a timed reading.
Upstream pin: bd92506447dc1333820cfe9a34ebbf497fe0219e

F3 is a research file format. The upstream project says not to use it in production. The Elixir library in this repository is an experiment on top of that prototype. It reads rows when the native decoder is compiled. Nothing in this article is a measurement of Elixir throughput.

Why look at another columnar format?

Parquet and ORC made it normal to move analytical data between engines. Both came out of the early 2010s, when the systems around them were Hive, Impala, and early Spark. They have been extended since. The awkward part is not that the specifications froze. It is that a specification change does not arrive in every reader at once. The F3 paper's look at a few hundred public Parquet files found that most of them still use a small, old subset of the format. A new encoding or a new timestamp unit is only useful if the reader on the other side of the exchange understands it.

That is a real constraint. It is not a reason to talk as if Parquet has stopped working. It is a reason the F3 authors tried a different split: keep a stable way to find bytes in the file, and let an unfamiliar encoding carry its own decoder.

What F3 is trying to be

The paper states three goals: interoperability, extensibility, and efficiency. A file is self-describing when a reader can find the schema, the locations of the columns, and, when the encoding is not one the reader was built with, a decoder that ships with the file. F3 puts the data, the metadata, and optional WebAssembly decoders in the same file. The paper reports that those binaries are on the order of kilobytes, and that in the authors' own experiments Wasm decoding was about 10–30% slower than a native decoder. Those numbers belong to that paper and that setup. This library has not reproduced them.

The format still uses a PAX layout, the same broad idea as Parquet and ORC: columns are stored together so a query can skip the ones it does not need. The authors' extra moves are about the units inside a column.

An IOUnit is a physical chunk of bytes the writer flushes. Its size is a storage choice. A spinning disk, a large SSD, and an object store do not want the same read size, and that choice should not have to be the same as the logical row group.

An EncUnit is a run of values that share one encoding. The default in the pinned writer is 65536 values, and the encoded size of that run can vary. IOUnits and EncUnits are different things. One IOUnit can hold several EncUnits. The schema still calls an IOUnit a Chunk in places. That name is historical.

A row group is the logical horizontal stripe. The pinned writer defaults to one row group per file and can be told to start another after a row threshold.

Encodings in the pinned schema are CASCADE (0), implemented with Vortex, and CUSTOM_WASM (255), which must be decoded by a Wasm binary stored in the file. A reader is not required to execute that binary. This library executes it only when a read passes wasm: :sandbox.

flowchart TB
  file[F3 file]
  file --> data[Row groups]
  file --> wasm[Optional Wasm binaries]
  file --> colmeta[Column metadata]
  file --> footer[Footer FlatBuffers]
  file --> post[Postscript and magic F3]
  data --> iounit[IOUnit]
  iounit --> encunit[EncUnit]

The footer points at column metadata. It is not a copy of every IOUnit header. The postscript is 32 little-endian bytes at the end of the file, and the last two of those bytes are the ASCII magic F3.

What the Elixir library is for

The library is a separate project. It uses the module F3 and, if it is ever published, the Hex name f3. On 2026-11-03 that Hex name was unused. Hex has not been published. The library pins one upstream commit and wire version 0.1. Opening a file acquires a byte source. Schema and footer metadata are decoded from the tail. Asking for rows returns columnar batches when the native decoder is compiled, and :native_unavailable when it is not.

From the repository root:

mix run examples/tutorial_01.exs

The script prints the pinned wire version. Reading and writing are implemented when the native build is compiled. Wasm stays off unless a read passes wasm: :sandbox.

Glossary

TermMeaning in the pinned sources
Postscript32-byte tail. Sizes, checksums, version, then F3. Not FlatBuffers.
FooterFlatBuffers root table. Arrow IPC schema and pointers.
Row groupLogical set of rows.
IOUnitPhysical write unit. Called Chunk in File.fbs.
EncUnitValues that share one encoding, contiguous inside an IOUnit.
CASCADEEncoding type 0. Vortex in the upstream writer.
CUSTOM_WASMEncoding type 255. Decode with the file's Wasm binary.

What is experimental?

Upstream F3 at bd92506447dc1333820cfe9a34ebbf497fe0219e is a research prototype. The schema file says the definition can change. The Elixir code decodes the footer. When F3_BUILD_NATIVE=1 it also decodes the CASCADE payloads in the committed fixtures and can run Wasm only for wasm: :sandbox. Wire versions other than 0.1 are rejected. There are no Elixir performance numbers in this article.

Next

Anatomy of an F3 file walks the same layout from the magic backward, using a file the pinned writer actually produced.

Sources