F3.Schema (F3 v0.1.0)

Logical Arrow schema read from an F3 footer.

A field type is a nested term. Primitive widths that the committed fixtures use are {:int, width, signed?} and {:float, precision}. {:list, field} and {:struct, fields} carry their children. Any other Arrow type id is {:unsupported, id} and does not by itself reject the file.

Row values are not part of this struct. F3.read/2 returns batches whose columns use these field names.

Summary

Types

arrow_type()

field()

One logical field.

t()

fields is the top-level schema, in footer order.

Functions

field_map(field)

JSON-ready form of one field. Keys are strings. A nested type is a map. A primitive type is the same string as type_label/1.

type_label(arg1)

Short type name used by F3.inspect/2.

Types

arrow_type()

@type arrow_type() ::
  :null
  | :bool
  | :utf8
  | :binary
  | {:int, pos_integer(), boolean()}
  | {:float, :half | :single | :double}
  | {:list, field()}
  | {:struct, [field()]}
  | {:unsupported, non_neg_integer()}

field()

@type field() :: %{name: String.t(), type: arrow_type(), nullable: boolean()}

One logical field.

name is the footer name. type is the nested Arrow term. nullable is true when the field accepts nulls. A writer keyword :int64 becomes nullable: false. {:int64, null: true} becomes nullable: true.

t()

@type t() :: %F3.Schema{fields: [field()]}

fields is the top-level schema, in footer order.

Functions

field_map(field)

@spec field_map(field()) :: map()

JSON-ready form of one field. Keys are strings. A nested type is a map. A primitive type is the same string as type_label/1.

Examples

iex> F3.Schema.field_map(%{name: "id", type: {:int, 64, true}, nullable: false})
%{"name" => "id", "nullable" => false, "type" => "int64"}

type_label(arg1)

@spec type_label(arrow_type()) :: String.t()

Short type name used by F3.inspect/2.

{:int, width, true} is int{width}. {:int, width, false} is uint{width}. Float precisions are float16, float32, and float64. Lists and structs include their children. This does not raise.

Examples

iex> F3.Schema.type_label({:int, 64, true})
"int64"

iex> F3.Schema.type_label({:list, %{name: "item", type: :utf8, nullable: true}})
"list<utf8>"