If you’ve encountered the dreaded RecordTooLargeException or The message is too large error in Apache Kafka, you’re not alone. This common issue can break data pipelines and confuse new users — but the fix is usually straightforward.
In this blog post, we’ll walk through why this happens and how to solve it step-by-step from both producer and broker sides.
🧠 Why This Happens
Kafka imposes limits on message size at several layers:
- Producer config:
max.request.size - Broker config:
message.max.bytes - Topic config:
max.message.bytes
When a message exceeds any of these thresholds, Kafka throws a RecordTooLargeException.
🛠️ Step-by-Step Solution
1. Increase Producer Max Request Size
Update your producer client config:
javaCopyEditProperties props = new Properties();
props.put("max.request.size", 10485760); // 10MB
Or in a YAML config (if using Kafka Connect):
max.request.size: 10485760
Tip: Start with 5-10 MB and increase only if necessary.
2. Check and Update Broker Config
Edit your server.properties on the Kafka broker:
propertiesCopyEditmessage.max.bytes=10485760
This must be equal or higher than the producer’s max.request.size.
Don’t forget to restart the broker after changing this.
3. Update Topic Config (Optional)
If a specific topic needs larger messages:
kafka-configs.sh --bootstrap-server <broker>:9092 \
--entity-type topics --entity-name your-topic \
--alter --add-config max.message.bytes=10485760
This allows flexibility: most topics can stay small, while selected ones allow big payloads.
4. Compress Your Messages
Compression drastically reduces message size. Supported algorithms:
- gzip
- snappy
- lz4
- zstd (Kafka 2.1+)
Producer example:
props.put("compression.type", "snappy");
Note: Compression happens on the client-side.
Common Pitfalls
- Setting broker values lower than the producer’s leads to failures.
- Don’t forget about schema overhead (e.g., in Avro/Protobuf).
- Monitor broker memory and disk — large messages can affect performance.
🔍 Useful Metrics to Monitor
BytesInPerSec/BytesOutPerSec(per topic)RequestSizes- Broker heap usage
Tools like Prometheus + Grafana can help visualize these.
Conclusion
Kafka is highly performant — but large messages need careful handling. If you follow the steps above and set proper limits, your pipeline will run smoothly without bloating or crashing.
If you’re constantly sending large payloads, consider:
- Breaking messages into chunks
- Using Kafka Streams or ksqlDB for transformation
- Offloading binary data (e.g., to S3) and sending only metadata in Kafka