cppkafka: high level C++ wrapper for rdkafka

cppkafka allows C++ applications to consume and produce messages using the Apache Kafka protocol. The library is built on top of librdkafka, and provides a high level API that uses modern C++ features to make it easier to write code while keeping the wrapper’s performance overhead to a minimum.

Features

• cppkafka is a high level C++ wrapper for rdkafka, aiming to allow using rdkafka in a simple, less error prone way.

• cppkafka provides an API to produce messages as well as consuming messages, but the latter is only supported via the high level consumer API. cppkafka requires rdkafka >= 0.9.4 in order to use it. Other wrapped functionalities are also provided, like fetching metadata, offsets, etc.

• cppkafka provides message header support. This feature requires rdkafka >= 0.11.4.

• cppkafka tries to add minimal overhead over librdkafka. A very thin wrapper for librdkafka messages is used for consumption so there’s virtually no overhead at all.

It’s simple!

cppkafka’s API is simple to use. For example, this code creates a producer that writes a message into some partition:

#include <cppkafka/cppkafka.h>

using namespace std;
using namespace cppkafka;

int main() {
    // Create the config
    Configuration config = {
        { "metadata.broker.list", "127.0.0.1:9092" }
    };

    // Create the producer
    Producer producer(config);

    // Produce a message!
    string message = "hey there!";
    producer.produce(MessageBuilder("my_topic").partition(0).payload(message));
    producer.flush();
}

Compiling

In order to compile cppkafka you need:

• librdkafka >= 0.9.4
• CMake >= 3.9.2
• A compiler with good C++11 support (e.g. gcc >= 4.8). This was tested successfully on g++ 4.8.3.
• The boost library (for boost::optional)

Now, in order to build, just run:

mkdir build
cd build
cmake <OPTIONS> ..
make
make install

CMake options

The following cmake options can be specified:

• RDKAFKA_ROOT : Specify a different librdkafka install directory.
• RDKAFKA_DIR : Specify a different directory where the RdKafkaConfig.cmake is installed.
• BOOST_ROOT : Specify a different Boost install directory.
• CPPKAFKA_CMAKE_VERBOSE : Generate verbose output. Default is OFF.
• CPPKAFKA_BUILD_SHARED : Build cppkafka as a shared library. Default is ON.
• CPPKAFKA_DISABLE_TESTS : Disable build of cppkafka tests. Default is OFF.
• CPPKAFKA_DISABLE_EXAMPLES : Disable build of cppkafka examples. Default is OFF.
• CPPKAFKA_BOOST_STATIC_LIBS : Link with Boost static libraries. Default is ON.
• CPPKAFKA_BOOST_USE_MULTITHREADED : Use Boost multi-threaded libraries. Default is ON.
• CPPKAFKA_RDKAFKA_STATIC_LIB : Link to Rdkafka static library. Default is OFF.
• CPPKAFKA_CONFIG_DIR : Install location of the cmake configuration files. Default is lib/cmake/cppkafka.
• CPPKAFKA_PKGCONFIG_DIR : Install location of the .pc file. Default is share/pkgconfig.
• CPPKAFKA_EXPORT_PKGCONFIG : Generate cppkafka.pc file. Default is ON.
• CPPKAFKA_EXPORT_CMAKE_CONFIG : Generate CMake config, target and version files. Default is ON.

Example:

cmake -DRDKAFKA_ROOT=/some/other/dir -DCPPKAFKA_BUILD_SHARED=OFF ...

Using

If you want to use cppkafka, you’ll need to link your application with:

• cppkafka
• rdkafka

If using CMake, this is simplified by doing:

find_package(CppKafka REQUIRED)

target_link_libraries(<YourLibrary> CppKafka::cppkafka)

Documentation

You can generate the documentation by running make docs inside the build directory. This requires Doxygen to be installed. The documentation will be written in html format at <build-dir>/docs/html/.

Make sure to check the wiki which includes some documentation about the project and some of its features.