I have developed a simple REST API to send the incoming message to Apache Kafka.
I have used Docker Kafka (https://github.com/spotify/docker-kafka) and the Docker Maven Plugin(https://github.com/fabric8io/docker-maven-plugin) to do this.
So before going through this post be familiarize yourself with Docker and Docker Compose
Docker Maven Plugin[Docker Maven Plugin] provides us a nice way to specify multiple images in POM.xml and link it as necessary. We can also use Docker compose for doing this. But I have used this plugin here.
- Clone the project (https://github.com/dkbalachandar/kafka-message-sender)
- Then go into kafka-message-sender folder
- Then enter ‘mvn clean install’
- Then enter ‘mvn docker:start’. Then enter ‘docker ps’ and make sure that there are two containers are running. The name of those containers are kafka, kafka-rest
- Then access http://localhost:8080/api/kafka/send/test (POST) and confirm that you see message has been sent on the browser
- Then enter the below command and make sure that whatever message which you sent is available at Kafka[Kafka Command Line Consumer] or you can also consume via a Flume agent[Kafka Flume Agent Consumer]
docker exec -it kafka /opt/kafka_2.11-0.8.2.1/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
We can verify the kafka topic properties by running the below command in the installation directory of Kafka
bin/kafka-topics.sh --zookeeper zookeeper.host.name:2181 --topic topic1 --describe
So you will have something like below,
Topic:topic1 PartitionCount:1 ReplicationFactor:3 Configs:
Topic: topic1 Partition: 0 Leader: 100 Replicas: 100,101,102 Isr: 100,101,102
So with that information, we can identify that there the replication factor for this topic is 3 and replicas broker node ids.
If we want to verify the same in Zookeeper then follow the below steps,
Login into the Zookeeper node and go to the installation directory and go into bin folder.
Then run zkCli as below
Then Enter the below command and see all the broker nodes id details
Then Enter the below command to know the details about the topic
Please refer below link for more commands, https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper