We recently faced an issue while testing Kafka functionality in our application. The producer application produced the message but it was not consumed by Ruby test case but the message was still available when we ran the command line utility.
we did further analysis and found out that both consumer application and test case used the same group.id. So as per the Kafka behavior, the message is getting consumed by one consumer with in a same consumer group. Please refer the below snippet from the Kafka documentation.
If all the consumer instances have the same consumer group, then this works just like a traditional queue balancing load over the consumers.
If all the consumer instances have different consumer groups, then this works like publish-subscribe and all messages are broadcast to all consumers.
If we want to consume the messages by multiple consumers, then we should have a different group.id specified in while configuraing the kafka consumer. We did this and it resolved the issue