Connectors Overview
Source and sink connectors for Kafka, Parquet, S3, Iceberg, and more.
Overview
Connectors live in krishiv-connectors. They implement the Source and Sink traits and are registered with a session via register_* methods or SQL DDL (CREATE SOURCE / CREATE SINK). Each connector carries its own Cargo feature gate.
Source Connectors
| Connector | Feature | SQL DDL Object |
|---|---|---|
| Kafka (Confluent or Apache) | kafka | CREATE SOURCE … TYPE KAFKA |
| Parquet (local / S3 / ADLS) | Always available | CREATE SOURCE … TYPE PARQUET |
| CSV / NDJSON | Always available | CREATE SOURCE … TYPE CSV |
| Iceberg (REST catalog) | iceberg | CREATE SOURCE … TYPE ICEBERG |
| Delta Lake | delta | CREATE SOURCE … TYPE DELTA |
| Hudi | hudi | CREATE SOURCE … TYPE HUDI |
| Arrow Flight | flight-sql | Registered programmatically |
Sink Connectors
| Connector | Feature | SQL DDL Object |
|---|---|---|
| Parquet (local / S3) | Always available | CREATE SINK … TYPE PARQUET |
| CSV / NDJSON | Always available | CREATE SINK … TYPE CSV |
| Kafka | kafka | CREATE SINK … TYPE KAFKA |
| Iceberg | iceberg | CREATE SINK … TYPE ICEBERG |
| Cassandra | cassandra | Programmatic only |
| Elasticsearch | elasticsearch | Programmatic only |
| HBase | hbase | Programmatic only |
| Vector stores | vector-sinks | Programmatic only |
Delivery Guarantees
The effective delivery guarantee is the weakest guarantee supported by the source, sink, and durability profile combination:
| Guarantee | Requirement |
|---|---|
| Best effort | Default — no special source or sink requirements. |
| At-least-once | Source must support offset/position tracking. Requires single-node-durable profile or higher. |
| Effectively-once | Idempotent/key-based sink. At-least-once source. Duplicate writes converge on one result. |
| Exactly-once | Certified source + transactional sink + distributed-durable profile. Source position and sink commit are coordinated by checkpoint protocol. |