Batch SQL
DataFusion-backed SQL over Apache Arrow RecordBatches and registered sources.
Engine maturity / Evidence map
Capability status, backed by the source tree.
Labels describe implementation maturity in the current Engine repository. They are not production-readiness promises, support tiers, or hosted-service SLAs.
Current mapPre-release
Four labels. One rule: verify the exact path.
- Available6 capability groups
- Preview11 capability groups
- Experimental2 capability groups
- Planned2 capability groups
Counts organize this page; they are not a quality score.
Current Engine map
Presence and stability are different things.
A capability can be implemented while its compatibility, recovery behavior, or operating envelope remains unsettled.
01 / Status
Available
Implemented in the source workspace. The project is still a developer preview, so this is not a production-readiness promise.
Apache Arrow data model
RecordBatch is the primary in-memory and IPC columnar format.
Rust Session / DataFrame API
Session, DataFrame, and Stream types are the primary Rust-facing API surface.
DataFusion SQL planning
SQL parsing, logical planning, expression evaluation, and local execution via DataFusion.
Embedded runtime mode
Runs all components in-process; no network endpoints required. Used in tests and local API calls.
Single-node runtime mode
Places core Engine components on one host. Durability depends on the configured state, checkpoint, and storage backends.
02 / Status
Preview
Implementation paths exist, but APIs, recovery behavior, or end-to-end combinations still need certification.
Stateful streaming
Streaming sessions, windows, state, and job APIs exist. Recovery and connector guarantees remain combination-specific.
Python bindings (source)
PyO3 bindings expose core Session and DataFrame APIs when built from the repository. Names and signatures can change before 1.0.
Distributed runtime mode
Remote coordinator and executor transport with bearer-token auth. Requires explicit Flight endpoint; no silent local fallback.
Iceberg integration
Iceberg-oriented catalog and table paths exist; backend and commit-protocol certification continues.
Kafka connector
Source and sink paths exist. Delivery guarantees depend on the exact source, sink, and checkpoint combination.
Parquet and object-store paths
Connector paths exist. The named S3 registry driver is currently local-backed; remote cloud construction is separately feature-gated.
Shuffle service
In-memory, local disk, object-store, and Flight-oriented shuffle paths behind the krishiv-shuffle crate API.
Checkpoint storage
Async checkpoint primitives with sync compatibility wrappers. Scheduler gRPC checkpoint acks use the async path.
State management
In-memory and RocksDB-backed keyed state, TTL, migration, and incremental state behind the krishiv-state crate API.
Kubernetes operator / CRD
CRD and operator integration in the krishiv-operator crate. Manifests live in k8s/.
Scheduler foundations
Job/task lifecycle, metadata-store, and leadership-coordination code exists; operational fault-tolerance certification is still in progress.
03 / Status
Experimental
Implemented and functional. APIs and semantics may change. Not certified for production use.
Delta Batch / IVM
DeltaBatch (weighted Arrow rows) and IncrementalFlow (view maintenance across ticks) are implemented with partitioning, snapshots, and checkpoint hooks. Distributed executor-side IVM execution is deferred.
Python connector surface
The current extension compiles several connector families by default; additional Cargo features add more. Packaging and API boundaries are not yet stable.
04 / Status
Planned
On the roadmap but not yet implemented. Do not rely on these without maintainer confirmation.
Distributed IVM
Executor-side incremental view maintenance across a distributed cluster. Requires distributed IVM protocol design.
Broad exactly-once certification
Engine does not claim exactly-once across arbitrary source, sink, state, and checkpoint combinations.
How to use this page
Evaluate a combination, not a badge.
Runtime mode, connector, checkpoint storage, sink semantics, and the pinned commit all influence the real boundary.
01Scope
Read the exact surface.
A label for Engine does not automatically promote every connector, deployment mode, or language binding.
02Evidence
Reproduce the path.
Prefer public APIs, CLI behavior, checked-in contracts, examples, and tests over dependency presence or roadmap copy.
03Change
Pin the commit.
Pre-release APIs and protocols can change. Record the revision and build features used by an evaluation.
Separate product status
Platform is coming soon and is not part of this matrix.
Its documentation remains an availability notice until a public preview creates real setup and API contracts.