Causes and remedies of poison pill in Apache Kafka

A poison pill is a message deliberately sent to a Kafka topic, designed to consistently fail when consumed, regardless of the number of consumption attempts. Poison Pill scenarios are frequently underestimated and can arise if not properly accounted for. Neglecting to address them can result in severe disruptions to the seamless operation of an event-driven system.

The poison pill for various reasons:

• The failure of deserialization of the consumed bytes from the Kafka topic on the consumer side.
• Incompatible serializer and deserializer between the message producer and consumer
• Corrupted records
• Data/Message was still being produced to the same Kafka topic even if the producer altered the key or value serializer.
• A different producer began publishing messages to the Kafka topic using a different key or value serializer.
• The consumer configured the wrong key or value deserializer which is not at all compatible with the serializer on the message producer side.

The consequences of poison pills if not handled properly:

• Consumer shutdown. When a consumer receives a poison pill message from the topic, it stops processing and terminates.
• If we surround the message consumption code with a try/catch block inside the consumer, log files get flooded with error messages and stack traces and eventually excessive disk space consumption on the system or nodes in the cluster.
• The poison pill message will block the partition of the topic, stopping the processing of any additional messages. As a result, the processing of the message will be tried again, most likely extremely quickly, placing a heavy demand on the system’s resources.

To prevent poison messages in Apache Kafka, we need to design our Kafka consumer application and Kafka topic-handling strategy in a way that can handle and mitigate the impact of problematic or malicious messages.

• Proper Serialization: Use a well-defined and secure serialization format for your messages, such as Avro or JSON Schema. This can help prevent issues related to the deserialization of malformed or malicious messages by consumers.
• Message Validation: Ensure that messages being produced to Kafka topics are validated to meet expected formats and constraints before they are published. This can be done by implementing strict validation rules or schemas for the messages. Messages that do not conform to these rules should be rejected at the producer level.
• Timeouts and Deadlines: Set timeouts and processing deadlines for your consumers. If a message takes too long to process, consider it a potential issue and handle it accordingly. This can help prevent consumers from getting stuck on problematic messages.
• Consumer Restart Strategies: Consider implementing strategies for automatically restarting consumers who encounter errors or become unresponsive. Tools like Apache Kafka Streams and Kafka Consumer Groups provide mechanisms for handling consumer failures and rebalancing partitions.
• Versioned Topics: When evolving your message schemas, create new versions of topics rather than modifying existing ones. This allows for backward compatibility and prevents consumers from breaking due to changes in message structure.
• When message loss is unacceptable, a code fix will be required to specifically handle the poison pill message. Besides, we can configure a dead letter queue (DLQ) and send the poison poll messages to it for retrying or analyzing the root cause.
• If the message or event loss is acceptable to a certain extent then by executing the built-in kafka-consumer-groups.sh script from the terminal, we can reset the consumer offset either to “–to-latest” or to a specific time. Thus by executing this, all the messages including the poison pill will be skipped that have not been consumed so far. But we need to make sure that the consumer group is not active otherwise, the offsets of a consumer or consumer group won’t be changed.

kafka-consumer-groups.sh –bootstrap-server localhost:9092 –group ourMessageConsumerGroup –reset-offsets –to-latest –topic myTestTopic –execute

or a specific time

kafka-consumer-groups.sh –bootstrap-server localhost:9092 –group ourMessageConsumerGroup –reset-offsets –to-datetime 2023-07-20T00:00:00.000 –topic myTestTopic –execute

Hope you have enjoyed this read. Please like and share if you feel this composition is valuable.

Written by
Gautam Goswami