01Available
Embedded
Runs Engine inside your process for local SQL, DataFrames, tests, and API evaluation. Batch results use Arrow RecordBatch values.
Engine architecture / Boundary map
One compute model. Three explicit placements.
Rust, Arrow, and DataFusion form the common execution spine. Embedded and single-node paths are available in the source workspace; distributed execution remains a Preview surface with a different operating boundary.
Placement mapExplicit
Topology never silently changes.
- EmbeddedAvailable · in process
- Single nodeAvailable · one host
- DistributedPreview · remote workers
- Incremental viewsExperimental · local first
A topology label does not imply durability, recovery, or connector guarantees.
01 / Placement
Choose a shape, then verify its maturity.
Placement is an explicit configuration decision. State, checkpointing, storage, and recovery remain separate choices.
Scroll horizontally to inspect the full diagram.
02Available
Single node
Places coordinator, executor, HTTP, and Flight boundaries on one host. Durability depends on the configured backends.
03Preview
Distributed
Coordinates work across remote executors. It is an evaluation path—not a promise of high availability, elastic scale, or production readiness.
02 / Request flow
A query moves through visible seams.
Batch begins with a session and DataFusion plan. Streaming and distributed paths add state and placement without changing the public entry point.
Scroll horizontally to inspect the full diagram.
Parse and bind
The session resolves tables, UDFs, expressions, and types against its catalog.
Plan and optimize
Logical work is rewritten and lowered to a physical plan; remote paths add task boundaries.
Execute
Arrow operators run the plan while state, shuffle, and checkpoint interfaces connect where required.
Return or continue
Batch returns RecordBatch values; streaming surfaces remain active through their job and stream interfaces.
03 / Cluster boundary
Control plane above. Data plane below.
The Preview distributed design separates lifecycle and placement decisions from execution. This is a boundary map, not an SLA.
Scroll horizontally to inspect the full diagram.
ControlCoordinator
Owns job and task lifecycle.
Scheduler and coordinator modules expose metadata, leadership, task assignment, and remote-control paths. Their presence does not certify a highly available control plane.
DataExecutors
Runs assigned fragments.
Executors connect work to shuffle, state, checkpoint, and transport interfaces. Failure behavior remains part of the distributed Preview boundary.
04 / Lifecycle
Submission and recovery are separate concerns.
Batch and streaming do not share identical recovery semantics. The complete source, state, checkpoint, and sink combination determines behavior.
Scroll horizontally to inspect the full diagram.
Validate
Resolve schema, types, and supported expressions before execution work is assigned.
Plan and place
Create executable fragments and assign them according to the selected runtime topology.
Run and observe
Execute the graph and evaluate failures against the configured state, checkpoint, source, and sink contracts.
05 / Honest boundaries
One architecture does not mean one maturity level.
Read capability status before treating an implementation path as a stable operating contract.
AvailableCurrent source
Batch and local placement
Batch SQL, DataFusion planning, Arrow data, embedded execution, single-node execution, and core source-built APIs.
PreviewCertification ongoing
Stateful and distributed paths
Streaming, remote execution, checkpoint and state integrations, and primary connector paths remain combination-specific.
ExperimentalMay change
Incremental view maintenance
Weighted deltas and IncrementalFlow exist as local-first evaluation surfaces, with distributed IVM still deferred.
Go deeper
Use the docs for contracts, not the diagram alone.
Start with execution modes, then inspect the operational page for the topology you intend to evaluate.