Pipeline DDL

CREATE SOURCE, CREATE SINK, and START PIPELINE for named connectors.

Overview

Pipeline DDL lets you register named sources and sinks using SQL and then start a pipeline that connects them. The source and sink registrations are stored in the PipelineRegistry and can be inspected or restarted without re-specifying connection details.

Statements

Statement	Description
`CREATE SOURCE <name> FROM <connector> OPTIONS (...)`	Register a named data source.
`CREATE SINK <name> INTO <connector> OPTIONS (...)`	Register a named data sink.
`DROP SOURCE <name>`	Remove a registered source.
`DROP SINK <name>`	Remove a registered sink.
`START PIPELINE <source> TO <sink>`	Start a streaming pipeline connecting source to sink.

Kafka Example

CREATE EXTERNAL TABLE orders (
  order_id BIGINT,
  customer_id BIGINT,
  amount DOUBLE,
  ts TIMESTAMP
) STORED AS KAFKA
LOCATION 'orders-topic'
OPTIONS (
  'bootstrap.servers' = 'kafka:9092',
  'group.id'          = 'krishiv-consumer'
);