Counting 3,146 Big Data & Machine Learning Frameworks, Toolsets, and Examples...
Suggestion? Feedback? Tweet @stkim1

Author
Last Commit
Sep. 23, 2018
Created
Sep. 19, 2012

librdkafka - the Apache Kafka C/C++ client library

Copyright (c) 2012-2018, Magnus Edenhill.

https://github.com/edenhill/librdkafka

Gitter chat Build status

librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

For an introduction to the performance and usage of librdkafka, see INTRODUCTION.md

See the wiki for a FAQ.

NOTE: The master branch is actively developed, use latest release for production use.

Overview

  • High-level producer
  • High-level balanced KafkaConsumer (requires broker >= 0.9)
  • Simple (legacy) consumer
  • Compression: snappy, gzip, lz4
  • SSL support
  • SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM) support
  • Broker version support: >=0.8 (see Broker version compatibility)
  • Stable C & C++ APIs (ABI safety guaranteed for C)
  • Statistics metrics
  • Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
  • RPM package: librdkafka and librdkafka-devel
  • Gentoo package: dev-libs/librdkafka
  • Portable: runs on Linux, OSX, Win32, Solaris, FreeBSD, AIX, ...

Language bindings

Users of librdkafka

Usage

Requirements

The GNU toolchain
GNU make
pthreads
zlib-dev (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)

Instructions

Building

  ./configure
  make
  sudo make install

NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.

NOTE: See CMake instructions for experimental CMake build (unsupported).

Usage in code

See examples/rdkafka_example.c for an example producer and consumer.

Link your program with -lrdkafka -lz -lpthread -lrt.

Documentation

The public APIs are documented in their respective header files:

To generate Doxygen documents for the API, type:

make docs

Configuration properties are documented in CONFIGURATION.md

For a librdkafka introduction, see INTRODUCTION.md

Examples

See the examples/sub-directory.

Tests

See the tests/sub-directory.

Support

File bug reports, feature requests and questions using GitHub Issues

Questions and discussions are also welcome on irc.freenode.org, #apache-kafka, nickname Snaps.

Commercial support

Commercial support is available from Edenhill services

Latest Releases
v0.11.6-RC3
 Sep. 17 2018
v0.11.6-RC2
 Sep. 7 2018
v0.11.6-RC1
 Sep. 6 2018
v0.11.6-PRE2
 Aug. 15 2018
v0.11.6-PRE1
 Aug. 13 2018