Kafka HTTP Gateway

Lightweight HTTP gateway exposing Kafka Producer, Consumer, and AdminClient APIs over HTTP. Built on pure Netty — no Spring, no heavy frameworks.

Features

• Producer — create, send (JSON & binary), partitions, transactions, lifecycle
• Consumer — create, poll, commit, seek, seek-to-beginning, seek-to-end, subscribe, assign, position, offsets, partitions, topics, lifecycle
• Admin — topics, configs, ACLs, consumer groups, offsets, SCRAM credentials, cluster info
• Binary protocol — high-performance binary format for producer send and consumer poll
• Optional Basic Auth — HTTP Basic Authentication via users.json

Requirements

• Java 21 (required)
• Kafka broker accessible from the gateway host

Build & Run

# Build
./gradlew build

# Run (default: 0.0.0.0:8086)
./gradlew run

# Create distribution
./gradlew installDist

# Run distribution
./build/install/kafka-gateway/bin/kafka-gateway

# Run distribution with custom host and port
KAFKA_GATEWAY_OPTS="-Dhost=127.0.0.1 -Dport=9090" ./build/install/kafka-gateway/bin/kafka-gateway

# Run tests
./gradlew test

Configuration

Deployment

systemd

A systemd unit file is provided in kafka-gateway.service:

sudo useradd -r -s /usr/sbin/nologin kafka-gateway
sudo mkdir -p /var/log/kafka-gateway
sudo chown kafka-gateway:kafka-gateway /var/log/kafka-gateway
sudo systemctl link /opt/kafka-gateway/kafka-gateway.service
sudo systemctl enable --now kafka-gateway

The service runs as user kafka-gateway from /opt/kafka-gateway/ with:

• Default bind address, port, and log directory configured via KAFKA_GATEWAY_OPTS (-Dhost=127.0.0.1 -Dport=8086 -Dlog.dir=/var/log/kafka-gateway)
• Application log files roll daily in /var/log/kafka-gateway; console output goes to journald
• LimitNOFILE=65536
• TimeoutStopSec=120 (greater than the default 60-second application shutdown deadline)

Windows Service (Apache Procrun)

1. Install prunsrv.exe from the Apache Commons Daemon binaries.
2. Extract the distribution archive into the installation folder (C:\kafka-gateway in the example below).
3. Navigate to the installation folder.

Run the provided script from an elevated prompt:

powershell -ExecutionPolicy Bypass -File .\install-windows-service.ps1 -InstallDir C:\kafka-gateway

To upgrade, replace the distribution files and re-run the script: it detects the existing service and updates it, refreshing the classpath (which bakes in the jar file names).

Parameters:

Manage the service:

prunsrv //ES//kafka-gateway   # start
prunsrv //SS//kafka-gateway   # stop
prunsrv //DS//kafka-gateway   # uninstall

Manual configuration (GUI): the Commons Daemon binaries also include prunmgr.exe:

prunmgr.exe //ES//kafka-gateway

The dialog edits the service parameters (startup, JVM, paths, logging) and can start/stop the service. Other modes: //MS// (tray monitor), //MR// (monitor and start the service if it is not running), //MQ// (quit all monitors). Changes are stored in the registry under HKLM\SOFTWARE\WOW6432Node\Apache Software Foundation\Procrun 2.0\kafka-gateway.

TLS

TLS is not implemented in the application. Use an NGINX reverse proxy for TLS termination (LAN-only deployment).

Authentication

Optional HTTP Basic Authentication via users.json in the working directory:

[
  {
    "username": "admin",
    "password": "secret"
  }
]

If users.json is missing, authentication is disabled and a warning is logged at startup.

API Overview

All endpoints accept JSON by default. Producer send and consumer poll also support a binary protocol for high-throughput scenarios.

The API is documented as an OpenAPI 3.1 specification in docs/api/openapi.yaml (see docs/api/README.md for Swagger UI setup).

Producer

Consumer

Admin

Management

Cluster

Topics

Configs

SCRAM

ACLs

Producers

Groups

Offsets

System

Instance Lifecycle

Each create request returns a string token — proof of ownership. All subsequent operations on that instance require the token.

Instances auto-expire after expirationTimeout (configurable per instance, 1 second to 24 hours). The expiration timer is reset on every request to the instance (poll, send, seek, describe, etc.). Use /touch when you need to keep an instance alive without performing any operation.

License

Apache License 2.0. See LICENSE and NOTICE.