KafkaAdapter is an experimental feature, changes in public API and usage are expected.
For instructions on downloading and building Calcite, start with thetutorial.
The Kafka adapter exposes an Apache Kafka topic as a STREAM table, so it can be queried using Calcite Stream SQL. Note that the adapter will not attempt to scan all topics, instead, users need to configure tables manually, one Kafka stream table is mapping to one Kafka topic.
A basic example of a model file is given below:
As Kafka message is schemaless, a KafkaRowConverter is required to specify row schema explicitly(with parameter
row.converter), and how to decode Kafka message to Calcite row. KafkaRowConverterImpl is used if not provided;
More consumer settings can be added in parameter
Assuming this file is stored as
kafka.model.json, you can connect to Kafka via
sqlline as follows:
sqlline will now accept SQL queries which access your Kafka topics.
With the Kafka table configured in the above model. We can run a simple query to fetch messages:
Kafka table is a streaming table, which runs continuously.
If you want the query to end quickly, add
LIMIT as follows: