How to optimize your Kafka producer for throughput

This tutorial requires access to an Apache Kafka cluster, and the quickest way to get started free is on Confluent Cloud, which provides Kafka as a fully managed service.

Make a local directory anywhere you’d like for this project:

Next, create a directory for configuration data:

From the Confluent Cloud Console, navigate to your Kafka cluster. From the Clients view, get the connection information customized to your cluster.

Create new credentials for your Kafka cluster, and then Confluent Cloud will show a configuration similar to below with your new credentials automatically populated (make sure show API keys is checked). Copy and paste it into a configuration/ccloud.properties file on your machine.

This tutorial has some steps for Kafka topic management and/or reading from or writing to Kafka topics, for which you can use the Confluent Cloud Console or install the Confluent CLI. Instructions for installing Confluent CLI and configuring it to your Confluent Cloud environment is available from within the Confluent Cloud Console: navigate to your Kafka cluster, click on the CLI and tools link, and run through the steps in the Confluent CLI tab.

In this step we’re going to create a topic for use during this tutorial. Use the following command to create the topic:

This creates a topic called topic-perf with a default number of 6 partitions. A topic partition is the unit of parallelism in Kafka, and messages to different partitions can be sent in parallel by producers, written in parallel by different brokers, and read in parallel by different consumers. In general, a higher number of topic partitions results in higher throughput, and to maximize throughput, you want enough partitions to distribute them across the brokers in your cluster. Although it might seem tempting just to create topics with a very large number of partitions, there are trade-offs to increasing the number of partitions. Choose the partition count carefully after benchmarking producer and consumer throughput in your environment. Also take into consideration the design of your data patterns and key assignments so that messages are distributed as evenly as possible across topic partitions to avoid a partition imbalance.

Run a performance test to capture a baseline measurement for your Kafka producer, using default configuration parameters. This test will send 10000 records of size 8000 bytes each.

Your results will vary depending on your connectivity and bandwidth to the Kafka cluster.

The key result to note is in the last line: the throughput being 134.560525 records/sec (1.03 MB/sec). This is the baseline producer performance with default values.

Run the Kafka producer performance test again, sending the exact same number of records of the same size as the previous test, but this time use configuration values optimized for throughput.

Here are some producer configuration parameters you can set to increase throughput. The values shown below are for demonstration purposes, and you will need to further tune these for your environment.

For a detailed explanation of these and other configuration parameters, read these recommendations for Kafka developers.

Your results will vary depending on your connectivity and bandwidth to the Kafka cluster.

The key result to note is in the last line: the throughput being 740.960285 records/sec (5.65 MB/sec). For the test shown here, 5.65 MB/sec throughput is about a 5x improvement over the 1.03 MB/sec baseline throughput, but again, the improvement factor will vary depending on your environment.

This tutorial has demonstrated how to get started with improving the producer throughput, and you should do further testing in your environment. Continue to tune these configuration parameters, and test it with your specific Kafka producer, not just using kafka-producer-perf-test.

You may try another tutorial, but if you don’t plan on doing other tutorials, use the Confluent Cloud Console or CLI to destroy all the resources you created. Verify they are destroyed to avoid unexpected charges.