F3.Writer (F3 v0.1.0)

Local F3 writer.

open/2 checks the schema and options, then starts the out-of-process helper when F3_BUILD_NATIVE=1. Batches are sent one at a time. close/1 finishes the helper and renames a temp file onto the destination. A failed finish deletes the temp file and leaves an existing destination unchanged. The rename is local-filesystem behavior, not a remote write.

Without the native helper, a valid open/2 returns :not_implemented and creates no file. Wasm is not written.

Summary

Types

t()

One local writer.

Functions

close(writer)

Finishes the file and renames it onto the destination.

open(path, opts \\ [])

Opens a writer for path.

write(writer, batch)

Writes one %F3.Batch{} or a list of row maps.

Types

t()

@type t() :: %F3.Writer{
  closed: boolean(),
  path: Path.t(),
  port: port() | nil,
  schema: F3.Schema.t(),
  temp: Path.t() | nil
}

One local writer.

port is the helper port, or nil before it starts. schema is the accepted logical schema. path is the destination. temp is the file renamed onto path at close/1. closed is true after a successful close.

Functions

close(writer)

@spec close(t()) :: {:ok, t()} | {:error, F3.Error.t()}

Finishes the file and renames it onto the destination.

A second call returns the same closed handle. A value that is not a writer is :not_implemented.

Examples

iex> {:error, error} = F3.Writer.close(:nope)
iex> error.reason
:writer_close

open(path, opts \\ [])

@spec open(Path.t(), keyword()) :: {:ok, t()} | {:error, F3.Error.t()}

Opens a writer for path.

schema: is a field keyword list or an %F3.Schema{}. A bare :int64 is non-null. {:float64, null: true} is nullable. compression: is :uncompressed (the default), :zstd, or :lz4. row_group_rows: starts a new row group after at least that many rows. iounit_bytes: defaults to 8 MiB and is rejected above 64 MiB. checksum: true stores a per-IOUnit xxHash.

A missing schema, an unknown option, and an IOUnit above 64 MiB return an error and create no file. Without the native helper, a valid schema returns :not_implemented and also creates no file. These checks do not raise.

Examples

iex> {:error, error} = F3.Writer.open("unused.f3", [])
iex> error.message
"schema is required"

iex> {:error, error} = F3.Writer.open("unused.f3", schema: [id: :int64], source: :remote)
iex> error.message
"unknown option source"

iex> opts = [schema: [price: {:float64, null: true}], iounit_bytes: 64 * 1024 * 1024 + 1]
iex> {:error, error} = F3.Writer.open("unused.f3", opts)
iex> error.category
:resource_limit

write(writer, batch)

@spec write(t(), F3.Batch.t() | [map()]) :: :ok | {:error, F3.Error.t()}

Writes one %F3.Batch{} or a list of row maps.

Row maps may use atom or string keys. A missing key is null only when the field is nullable. An integer outside the field width is :invalid_type. A binary that is not UTF-8 is :invalid_argument. An empty batch is accepted and does not call the helper. A value that is not a writer is :not_implemented. This does not raise.

Examples

iex> {:error, error} = F3.Writer.write(:nope, [])
iex> error.category
:not_implemented