docker-gc: Utility to do garbage collection

The “docker-gc” is used to clean up the unused containers and images. By defualt, It removes all the containers that exited more than an hour ago and also the images that don’t belong to any remaining containers.

Refer the GitHub(docker-gc)

We can use this utility as a script and container. To run this utility as container, then use the below command. The container will start up, run a garbage collection process, then shutdown.


docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc

There is an another cron utility docker-gc-cron which internally uses the docker-gc utilty for cleaning up the unused docker containers and images. This utility is very useful for scheduling the cleanup process on Jenkins CI nodes.

docker-slim: Utility to reduce the fat size docker images

The “docker-slim” is a magic diet pill for our containers. It uses the static and dynamic analysis to create skinny image variants of our fat images.

To use docker-slim, we have to download its binary from docker-slim. Binaries are available for Linux and Mac. Once we download the binary, then add it to PATH.

I have used the “helloworld-docker” image for this exercise.  Refer my GitHub project (java-rest-docker).

Assume that, I have already built the docker image. Let’s run the “docker images” command to check the size of it. The below is the response.

Command: docker images


REPOSITORY               TAG                 IMAGE ID            CREATED          SIZE
hello-rest               latest              81a33e78995c        3 minutes ago       651.4 MB
java                     8                   d23bdf5b1b1b        4 months ago        643.2 MB

Let’s run the docker-slim utility now. The utility will run and create a new slim image.


  sudo docker-slim build --http-probe hello-rest

Then run the “docker images” command and check the output. In the output, I see a new image named “hello-rest.slim” and its size is 193.6 MB which is better than the original image size. Make sure to run the newly created image and check if its working or not.

Command: docker images


REPOSITORY               TAG                 IMAGE ID            CREATED          SIZE
hello-rest.slim          latest              1507f864ebbc        9 seconds ago       193.6 MB
hello-rest-docker        latest              81a33e78995c        3 minutes ago       651.4 MB
java                     8                   d23bdf5b1b1b        4 months ago        643.2 MB

Create Docker image with Maven build

There are lots of maven Docker plugin available to integrate the docker with maven.

In this example, I am going to show how to build the Docker image while building a maven project.

Copy the below snippet and put into your pom.xml file and then create a maven property “docker.image.name” with the appropriate docker image name and also make sure that the Dockerfile available in the correct location.

Then run the ‘mvn install’ and once its done, run ‘docker images’ and check that the docker image is available in the list of images.

pom.xml:


  <plugin>
	<groupId>org.codehaus.mojo</groupId>		
	<artifactId>exec-maven-plugin</artifactId>		
	<version>1.4.0</version>		
	<executions>		
		<execution>		
			<goals>		
				<goal>java</goal>		
			</goals>		
		</execution>		
		<execution>		
			<id>build-image</id>		
			<phase>install</phase>		
			<goals>		
				<goal>exec</goal>		
			</goals>		
			<configuration>		
				<executable>docker</executable>		
				<arguments>		
					<argument>build</argument>		
					<argument>-t=${docker.image.name}</argument>		
					<argument>.</argument>		
				</arguments>		
			</configuration>		
		</execution>		
	</executions>			
 </plugin>		
<plugin>
      

Rest API to produce message to Kafka using Docker Maven Plugin

I have developed a simple REST API to send the incoming message to Apache Kafka.

I have used Docker Kafka (https://github.com/spotify/docker-kafka) and the Docker Maven Plugin(https://github.com/fabric8io/docker-maven-plugin) to do this.

So before going through this post be familiarize yourself with Docker and Docker Compose

Docker Maven Plugin[Docker Maven Plugin] provides us a nice way to specify multiple images in POM.xml and link it as necessary. We can also use Docker compose for doing this. But I have used this plugin here.

    1. Clone the project (https://github.com/dkbalachandar/kafka-message-sender)
    2. Then go into kafka-message-sender folder
    3. Then enter ‘mvn clean install’
    4. Then enter  ‘mvn docker:start’. Then enter ‘docker ps’ and make sure that there are two containers are running. The name of those containers are kafka, kafka-rest
    5. Then access http://localhost:8080/api/kafka/send?msg=test and confirm that you see message has been sent on the browser
    6. Then enter the below command and make sure that whatever message which you sent is available at Kafka[Kafka Command Line Consumer] or you can also consume via a Flume agent[Kafka Flume Agent Consumer]
docker exec -it kafka /opt/kafka_2.11-0.8.2.1/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Thread & Heap dumps From a Docker container

Follow the below steps to take the thread and Heap dumps from a docker container

1. Run the below command to bash into the container. Please change the CONTAINER_NAME appropriately

      docker exec -it CONTAINER_NAME bash
    


2. Then type jps to find the all the Java application details and extract the PID for your application

    jps
  

3. Then run the below command to get the thread dump. Please change the PID appropriately

     jstack PID > threadDump.tdump 
     

4. Then run the below command to get the Heap dump. Please change the PID appropriately

        jmap -dump:live,format=b,file=heapDump.hprof PID 
     

5. Then exit from the docker container and download the threadDump.tdump and heapDump.hprof from the docker container by running the below command. Please change the CONTAINER_NAME appropriately

      sudo docker cp CONTAINER_NAME:threadDump.tdump .
      sudo docker cp CONTAINER_NAME:heapDump.hprof .
    

Copy files from host to Docker container and vice versa

Docker command to copy/update the files from host to container

   Command: 
   docker cp LOCALFILE_WITH_PATH CONTAINER_NAME:DEST_PATH
   Example: 
   docker cp config.xml CONTAINER_NAME:/opt/app/config.xml

Docker command to copy the files from container to local host

   Command:
   docker cp CONTAINER_NAME:SOURCE_PATH LOCALDIRPATH
   Example:
   docker cp CONTAINER_NAME:/opt/app/config.xml .

Replace the CONTAINER_NAME appropriately

How to use Docker Maven plugin

Recently, I have explored on various Docker Maven plugins and used https://github.com/fabric8io/docker-maven-plugin as its somewhat easy. Please note that as of now, this does not have support to integrate with Docker compose file. But it provides a lot of configuration tags by using which we can completely replace all the configurations found in a docker compose file.

Assume that we have two Restful services running on port number 8080, 8081 respectively. Both are entirely different. The first Restful application runs on 8080 access the second Restful application running on port number 8081. Refer below the snippet of pom.xml which uses the docker maven plugin


<build>
    <plugins>
        <plugin>
            <groupId>io.fabric8</groupId>
            <artifactId>docker-maven-plugin</artifactId>
            <version>0.15.1</version>
            <configuration>
                <images>
                    <!-- Application 1 -->
                    <image>
                        <name>middleware-rest</name>
                        <alias>middleware-rest</alias>
                        <build>
                            <cleanup>true</cleanup>
                            <tags>
                                <tag>latest</tag>
                            </tags>
                            <ports>
                                <port>8080</port>
                            </ports>
                            <dockerFileDir>middleware-rest</dockerFileDir>
                            <assembly>
                                <mode>dir</mode>
                                <inline xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
                                    <id>middleware-rest</id>
                                    <files>
                                        <file>
                                            <source>${project.build.directory}/mwRest-${project.version}-appjar.jar</source>
                                            <outputDirectory>./</outputDirectory>
                                            <destName>mwRest.jar</destName>
                                        </file>
                                    </files>
                                </inline>
                            </assembly>
                        </build>
                        <run>
                            <namingStrategy>none</namingStrategy>
                            <ports>
                                <port>8080:8080</port>
                            </ports>
                            <wait>
                                <tcp>
                                    <ports>
                                        <port>8080</port>
                                    </ports>
                                </tcp>
                                <time>60000</time>
                            </wait>
                            <env>
                                <CLIENT_MW_URL>http://client-middleware-rest:8080</CLIENT_MW_URL>
                            </env>
                            <links>
                                <link>client-middleware-rest:client-middleware-rest</link>
                            </links>
                            <log>
                                <enabled>true</enabled>
                                <color>red</color>
                                <driver>
                                    <name>json-file</name>
                                    <opts>
                                        <max-size>10m</max-size>
                                        <max-file>5</max-file>
                                    </opts>
                                </driver>
                            </log>
                        </run>
                    </image>
                    <!-- Application 2  -->
                    <image>
                        <name>client-middleware-rest</name>
                        <alias>client-middleware-rest</alias>
                        <build>
                            <cleanup>true</cleanup>
                            <tags>
                                <tag>latest</tag>
                            </tags>
                            <ports>
                                <port>8081</port>
                            </ports>
                            <dockerFileDir>client-middleware-rest</dockerFileDir>
                            <assembly>
                                <mode>dir</mode>
                                <inline xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
                                    <id>client-middleware-rest</id>
                                    <files>
                                        <file>
                                            <source>${project.build.directory}/clientmwRest-${project.version}-appjar.jar</source>
                                            <outputDirectory>./</outputDirectory>
                                            <destName>clientmwRest.jar</destName>
                                        </file>
                                    </files>
                                </inline>
                            </assembly>
                        </build>
                        <run>
                            <namingStrategy>none</namingStrategy>
                            <ports>
                                <port>8081:8081</port>
                            </ports>
                            <wait>
                                <tcp>
                                    <ports>
                                        <port>8081</port>
                                    </ports>
                                </tcp>
                                <time>60000</time>
                            </wait>
                            <log>
                                <enabled>true</enabled>
                                <color>red</color>
                                <driver>
                                    <name>json-file</name>
                                    <opts>
                                        <max-size>10m</max-size>
                                        <max-file>5</max-file>
                                    </opts>
                                </driver>
                            </log>
                        </run>
                    </image>
                </images>
            </configuration>
            <executions>
                <execution>
                    <id>start</id>
                    <phase>pre-integration-test</phase>
                    <goals>
                        <goal>build</goal>
                        <goal>start</goal>
                    </goals>
                </execution>
                <execution>
                    <id>stop</id>
                    <phase>post-integration-test</phase>
                    <goals>
                        <goal>stop</goal>
                    </goals>
                </execution>
                <execution>
                    <id>remove</id>
                    <phase>post-integration-test</phase>
                    <goals>
                        <goal>stop</goal>
                        <goal>remove</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

So once you have this configuration, then we can build the containers by running mvn docker:build

To start all the containers, run mvn docker:start
To stop all the containets, run mvn docker:remove

For further information, Please access https://github.com/fabric8io/docker-maven-plugin

Also refer my other post to know more about this Rest API to produce message to Kafka using Docker Maven Plugin