arrowstore
import "github.com/umbralcalc/stochadex/pkg/arrowstore"Package arrowstore is an opt-in, Apache Arrow-native storage + output function for the simulation output (egress) boundary. It is a SEPARATE Go module (github.com/umbralcalc/stochadex/pkg/arrowstore) so that Arrow and its dependency tree — which includes a gonum v0.17 requirement — stay entirely out of the core engine’s go.mod. The engine module remains lean and WASM-clean; you opt in only by importing this module.
What it is for: getting simulation output into Apache Arrow with minimal cost, so it can be handed to DuckDB / Polars / pandas (or written as Parquet/Feather) without a conversion pass. It is the egress foundation the analytical-sink integrations build on.
It is NOT a general-purpose faster in-memory store. Two honest results from its benchmark (Apple M4, reproduced by BenchmarkAppend / BenchmarkToArrow):
Append hot path, no materialisation: vs simulator.StateTimeStorage this trades a one-heap-allocation-per-row jagged [][][]float64 for a growing contiguous FixedSizeListBuilder. Allocation COUNT collapses to a constant in row count (a big GC-pressure win — e.g. ~50 allocs vs ~2000 at 2000 rows), but wall-clock is only comparable-to-better at wide state and is actually slower at mid widths, and transient memory is higher (builder capacity doubling over-allocates). So if all you want is [][]float64 in memory, keep simulator.StateTimeStorage — it is the right default.
Getting to Arrow (the reason this exists): this wins decisively — ~2.2–2.7× faster, far fewer allocations, and roughly half the memory of appending to StateTimeStorage and then converting [][]float64 → Arrow — because it builds the Arrow arrays directly.
So: reach for ArrowStateTimeStorage when the output is destined for the columnar/analytical world; keep the pure-Go StateTimeStorage otherwise.
Concurrency mirrors simulator.StateTimeStorage exactly and MUST be preserved:
- one builder per partition index, written only by that partition’s goroutine (lock-free);
- a single shared time column, deduplicated once globally under a mutex with an atomic fast path (so the common “same timestamp across partitions this step” case skips the lock).
Do not share a builder across goroutines, append out of the owning goroutine, or add a lock to the per-partition path — any of those trades the allocation win for contention.
WASM note: the engine is WASM-clean; this module is not automatically so, since Arrow uses assembly SIMD. Build it for WASM with Arrow’s noasm build tag if you need that target.
Index
- type
ArrowStateTimeStorage
- func NewArrowStateTimeStorage() *ArrowStateTimeStorage
- func (s *ArrowStateTimeStorage) AppendByIndex(index int, time float64, values []float64)
- func (s *ArrowStateTimeStorage) ArrayForIndex(index int) *array.FixedSizeList
- func (s *ArrowStateTimeStorage) Finalize()
- func (s *ArrowStateTimeStorage) GetNames() []string
- func (s *ArrowStateTimeStorage) GetTimes() []float64
- func (s *ArrowStateTimeStorage) GetValues(name string) [][]float64
- func (s *ArrowStateTimeStorage) IndexOf(name string) (int, bool)
- func (s *ArrowStateTimeStorage) PreRegisterPartitions(names []string)
- func (s *ArrowStateTimeStorage) Record() arrow.Record
- func (s *ArrowStateTimeStorage) Release()
- func (s *ArrowStateTimeStorage) TimeArray() *array.Float64
- type ArrowStateTimeStorageOutputFunction
type ArrowStateTimeStorage
ArrowStateTimeStorage is the Arrow-native, opt-in counterpart to simulator.StateTimeStorage. See the package doc for the rationale and the concurrency contract, both of which mirror simulator.StateTimeStorage.
type ArrowStateTimeStorage struct {
// contains filtered or unexported fields
}func NewArrowStateTimeStorage
func NewArrowStateTimeStorage() *ArrowStateTimeStorageNewArrowStateTimeStorage returns an empty storage backed by the default Go allocator.
func (*ArrowStateTimeStorage) AppendByIndex
func (s *ArrowStateTimeStorage) AppendByIndex(index int, time float64, values []float64)AppendByIndex appends one row for a pre-registered partition and records the time at most once per unique timestamp. Lock-free for the partition builder (one goroutine per index); the shared time column uses the same atomic-guarded dedup as simulator.StateTimeStorage.
func (*ArrowStateTimeStorage) ArrayForIndex
func (s *ArrowStateTimeStorage) ArrayForIndex(index int) *array.FixedSizeListArrayForIndex returns partition index’s rows as a FixedSizeList<float64>[width] Arrow array, or nil if the partition never produced a row. The storage retains ownership.
func (*ArrowStateTimeStorage) Finalize
func (s *ArrowStateTimeStorage) Finalize()Finalize materialises every builder into an immutable Arrow array. It is single-threaded, idempotent, and must be called after the simulation run (never concurrently with AppendByIndex). It consumes the builders. Call Release when done with the storage.
func (*ArrowStateTimeStorage) GetNames
func (s *ArrowStateTimeStorage) GetNames() []stringGetNames returns the registered partition names in index order.
func (*ArrowStateTimeStorage) GetTimes
func (s *ArrowStateTimeStorage) GetTimes() []float64GetTimes returns the deduplicated time axis as a plain slice, for compatibility with analysis code written against simulator.StateTimeStorage.
func (*ArrowStateTimeStorage) GetValues
func (s *ArrowStateTimeStorage) GetValues(name string) [][]float64GetValues returns partition name’s rows as [][]float64, for compatibility with analysis code written against simulator.StateTimeStorage. Each row shares the underlying Arrow buffer (read-only); copy if you need to mutate.
func (*ArrowStateTimeStorage) IndexOf
func (s *ArrowStateTimeStorage) IndexOf(name string) (int, bool)IndexOf returns the index and true if name is registered, or 0 and false.
func (*ArrowStateTimeStorage) PreRegisterPartitions
func (s *ArrowStateTimeStorage) PreRegisterPartitions(names []string)PreRegisterPartitions assigns a stable index and a (lazily-filled) builder slot to each name. Widths are learned from the first row each partition appends, so the fixed-width builder is created by the owning goroutine on its first AppendByIndex — safe because each index is written by exactly one goroutine.
func (*ArrowStateTimeStorage) Record
func (s *ArrowStateTimeStorage) Record() arrow.RecordRecord assembles a single Arrow record: the shared time column followed by one FixedSizeList column per partition. It is only well-defined when every partition produced the same number of rows as the time axis (the usual case: every partition outputs every step). It returns nil if that alignment does not hold. The caller owns the record (Release).
func (*ArrowStateTimeStorage) Release
func (s *ArrowStateTimeStorage) Release()Release frees the materialised Arrow arrays. Safe to call once after Finalize.
func (*ArrowStateTimeStorage) TimeArray
func (s *ArrowStateTimeStorage) TimeArray() *array.Float64TimeArray returns the shared time column as an Arrow array (Finalize is called if needed). The storage retains ownership; do not Release the returned array directly.
type ArrowStateTimeStorageOutputFunction
ArrowStateTimeStorageOutputFunction is the opt-in, Arrow-native counterpart to simulator.StateTimeStorageOutputFunction. It implements simulator.OutputFunction and writes each partition’s state into an ArrowStateTimeStorage at the egress boundary.
type ArrowStateTimeStorageOutputFunction struct {
Store *ArrowStateTimeStorage
// contains filtered or unexported fields
}func (*ArrowStateTimeStorageOutputFunction) Configure
func (f *ArrowStateTimeStorageOutputFunction) Configure(settings *simulator.Settings)Configure pre-registers all partition names on Store and caches their indices for lock-free lookup in Output. Safe to call multiple times.
func (*ArrowStateTimeStorageOutputFunction) Output
func (f *ArrowStateTimeStorageOutputFunction) Output(partitionName string, state []float64, cumulativeTimesteps float64)Output stores one partition’s state row.
Generated by gomarkdoc