Counting 3,663 Big Data & Machine Learning Frameworks, Toolsets, and Examples...
Suggestion? Feedback? Tweet @stkim1

Author
Last Commit
Mar. 26, 2019
Created
Sep. 27, 2016

KSQL rocket KSQL - Streaming SQL for Apache Kafka

KSQL is now GA and officially supported by Confluent Inc. Get started with KSQL today.

KSQL is the streaming SQL engine for Apache Kafka. It provides a simple and completely interactive SQL interface for stream processing on Kafka; no need to write code in a programming language such as Java or Python. KSQL is distributed, scalable, reliable, and real-time. It supports a wide range of powerful stream processing operations including aggregations, joins, windowing, sessionization, and much more. You can find more KSQL tutorials and resources here if you are interested.

Click here to watch a screencast of the KSQL demo on YouTube. KSQL screencast

Getting Started and Download

Stable Releases

Stable releases are published every four months and are officially supported by Confluent.

  1. Download latest stable KSQL, which is included in Confluent Platform.
  2. Follow the Quick Start.
  3. Read the KSQL Documentation, notably the KSQL Tutorials and Examples, which include Docker-based variants.

Preview Releases

In addition to supported stable KSQL releases, we also provide monthly preview releases. We encourage you to try them in development and testing environments and to take advantage of Confluent Community resources to get help and share feedback.

Documentation

See KSQL documentation for the latest stable release.

Use Cases and Examples

Streaming ETL

Apache Kafka is a popular choice for powering data pipelines. KSQL makes it simple to transform data within the pipeline, readying messages to cleanly land in another system.

CREATE STREAM vip_actions AS
  SELECT userid, page, action
  FROM clickstream c
  LEFT JOIN users u ON c.userid = u.user_id
  WHERE u.level = 'Platinum';

Anomaly Detection

KSQL is a good fit for identifying patterns or anomalies on real-time data. By processing the stream as data arrives you can identify and properly surface out of the ordinary events with millisecond latency.

CREATE TABLE possible_fraud AS
  SELECT card_number, count(*)
  FROM authorization_attempts
  WINDOW TUMBLING (SIZE 5 SECONDS)
  GROUP BY card_number
  HAVING count(*) > 3;

Monitoring

Kafka's ability to provide scalable ordered messages with stream processing make it a common solution for log data monitoring and alerting. KSQL lends a familiar syntax for tracking, understanding, and managing alerts.

CREATE TABLE error_counts AS
  SELECT error_code, count(*)
  FROM monitoring_stream
  WINDOW TUMBLING (SIZE 1 MINUTE)
  WHERE  type = 'ERROR'
  GROUP BY error_code;

Join the Community

You can get help, learn how to contribute to KSQL, and find the latest news by connecting with the Confluent community.

Contributing

Contributions to the code, examples, documentation, etc. are very much appreciated.

License

The project is licensed under the Confluent Community License.

Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation.

Latest Releases
v5.1.2-cp1-rc1
 Mar. 12 2019
v5.1.2-rc1
 Feb. 17 2019
v5.1.1-rc1
 Feb. 6 2019
v5.1.0-rc3
 Dec. 14 2018
v5.1.0-rc2
 Dec. 13 2018