Execution Model
RuntimeMode, ExecutionPlacement, and how plans move through Krishiv.
RuntimeMode vs ExecutionPlacement
RuntimeMode is the user-visible mode; ExecutionPlacement describes where data-plane work may actually run. They are intentionally separate.
| RuntimeMode | Description |
|---|---|
Embedded | In-process; no daemon. Best for tests and local API use. |
SingleNode | All engine pieces on one host; may use local daemon or in-process cluster. |
Distributed | Remote coordinator/executor transport. Requires an explicit Flight coordinator URL. |
Note: Distributed sessions must not silently fall back to in-process execution. An explicit
KRISHIV_COORDINATOR endpoint is required.Sync/Async Seam
The primary sync methods (collect_batch_sql, accept_plan) delegate to async variants (collect_batch_sql_async) via block_on at a single seam. Callers inside Tokio contexts should prefer async variants. Remote runtimes drive Flight/gRPC calls directly in the async path; in-process runtimes off-load DataFusion work to the blocking pool via spawn_blocking.
Durability Profiles
| Profile | State | Shuffle | Checkpoints |
|---|---|---|---|
dev-local | In-memory | In-memory | Ephemeral local; not restart-durable |
single-node-durable | RocksDB local | Local disk | Local filesystem; restart-durable on one host |
distributed-durable | RocksDB (restored from checkpoint) | Tiered: local + object store | Object store; etcd metadata; fenced coordination |
Delivery Guarantees
The end-to-end delivery guarantee is the weakest guarantee supplied by the source, sink, checkpoint storage, and selected durability profile.
- Best effort: failure can lose or duplicate records.
- At least once: acknowledged source positions are not advanced before durable output, but replay may duplicate output.
- Effectively once: deterministic/idempotent sink keys make repeated writes converge on one visible result.
- Exactly once: source position and sink publication are coordinated by checkpoint protocol and a transactional/two-phase sink. Requires certified source + sink + checkpoint combination.