F3.
Copy Markdown
Projection, row-group selection, and batching.
One decoder call decodes a whole encoding unit. Elixir then slices that
column to batch_size rows. The default batch size is 8192. The committed
fixtures are smaller than one unit, so a row group becomes one batch unless
batch_size is set lower. F3.stream/2 does not pread until the first pull.
Enum.take/2 stops further reads. The caller still owns F3.close/1.
Chunk checksums are verified only when the footer stored one. The eight
fixtures store none. Shared dictionaries are refused. CUSTOM_WASM is
refused unless the caller passes wasm: :sandbox or registers an
F3.Extension. Metadata inspection never executes a module.
Summary
Functions
Default number of rows in one batch.
Reads the selected row groups into columnar batches.
Refuses a shared dictionary. Wasm is decided later by F3.Dispatch.
A chunk with dictionary: :none and no refused encoding unit returns :ok.
A shared dictionary is :unsupported_encoding and does not raise.
Lazy batches. The first pull reads the footer. Later pulls read encoding
units. A pull that cannot decode throws {:f3, %F3.Error{}}. Without the
native decoder this returns the error and does not return a stream.
Functions
@spec default_batch_size() :: pos_integer()
Default number of rows in one batch.
Examples
iex> F3.Reader.default_batch_size()
8192@spec read(F3.File.t(), keyword()) :: {:ok, [F3.Batch.t()]} | {:error, F3.Error.t()}
Reads the selected row groups into columnar batches.
Options match F3.read/2. Without the native decoder this returns
:native_unavailable. Options that are not a keyword list are
:invalid_argument. This does not raise for those cases.
Examples
iex> {:ok, file} = F3.open("test/fixtures/small.f3")
iex> {:error, error} = F3.Reader.read(file, columns: ["symbol"], batch_size: 1024)
iex> error.category
:native_unavailable@spec reject_chunk(F3.RowGroup.chunk()) :: :ok | {:error, F3.Error.t()}
Refuses a shared dictionary. Wasm is decided later by F3.Dispatch.
A chunk with dictionary: :none and no refused encoding unit returns :ok.
A shared dictionary is :unsupported_encoding and does not raise.
Examples
iex> :ok = F3.Reader.reject_chunk(%{dictionary: :none, encunits: []})
iex> {:error, error} = F3.Reader.reject_chunk(%{dictionary: {:shared, 3}})
iex> error.message
"shared dictionary 3 is not loaded"@spec stream(F3.File.t(), keyword()) :: {:ok, Enumerable.t()} | {:error, F3.Error.t()}
Lazy batches. The first pull reads the footer. Later pulls read encoding
units. A pull that cannot decode throws {:f3, %F3.Error{}}. Without the
native decoder this returns the error and does not return a stream.
Examples
iex> {:ok, file} = F3.open("test/fixtures/small.f3")
iex> {:error, error} = F3.Reader.stream(file, [])
iex> error.category
:native_unavailable