Apache Kafka – Consumer Group

We recently faced an issue while testing Kafka functionality in our application. The producer application produced the message but it was not consumed by Ruby test case but the message was still available when we ran the command line utility.

we did further analysis and found out that both consumer application and test case used the same group.id. So as per the Kafka behavior, the message is getting consumed by one consumer with in a same consumer group. Please refer the below snippet from the Kafka documentation.


If all the consumer instances have the same consumer group, then this works just like a traditional queue balancing load over the consumers.

If all the consumer instances have different consumer groups, then this works like publish-subscribe and all messages are broadcast to all consumers.

If we want to consume the messages by multiple consumers, then we should have a different group.id specified in while configuraing the kafka consumer. We did this and it resolved the issue

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s