docker-gc: Utility to do garbage collection

The “docker-gc” is used to clean up the unused containers and images. By defualt, It removes all the containers that exited more than an hour ago and also the images that don’t belong to any remaining containers.

Refer the GitHub(docker-gc)

We can use this utility as a script and container. To run this utility as container, then use the below command. The container will start up, run a garbage collection process, then shutdown.


docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc

There is an another cron utility docker-gc-cron which internally uses the docker-gc utilty for cleaning up the unused docker containers and images. This utility is very useful for scheduling the cleanup process on Jenkins CI nodes.

docker-slim: Utility to reduce the fat size docker images

The “docker-slim” is a magic diet pill for our containers. It uses the static and dynamic analysis to create skinny image variants of our fat images.

To use docker-slim, we have to download its binary from docker-slim. Binaries are available for Linux and Mac. Once we download the binary, then add it to PATH.

I have used the “helloworld-docker” image for this exercise.  Refer my GitHub project (java-rest-docker).

Assume that, I have already built the docker image. Let’s run the “docker images” command to check the size of it. The below is the response.

Command: docker images


REPOSITORY               TAG                 IMAGE ID            CREATED          SIZE
hello-rest               latest              81a33e78995c        3 minutes ago       651.4 MB
java                     8                   d23bdf5b1b1b        4 months ago        643.2 MB

Let’s run the docker-slim utility now. The utility will run and create a new slim image.


  sudo docker-slim build --http-probe hello-rest

Then run the “docker images” command and check the output. In the output, I see a new image named “hello-rest.slim” and its size is 193.6 MB which is better than the original image size. Make sure to run the newly created image and check if its working or not.

Command: docker images


REPOSITORY               TAG                 IMAGE ID            CREATED          SIZE
hello-rest.slim          latest              1507f864ebbc        9 seconds ago       193.6 MB
hello-rest-docker        latest              81a33e78995c        3 minutes ago       651.4 MB
java                     8                   d23bdf5b1b1b        4 months ago        643.2 MB

Hadoop merge large part files with ‘getmerge’ command

I recently faced an issue while trying to merge large number of part files into a single file. I had used the ‘cat’ command for the merge. My command looks like below,


   hadoop fs -cat ${INPUT_PATH}/part* > ${OUTPUT_FOLDER}/output.xml

Assume that INPUT_PATH is the location of part files and OUTPUT_FOLDER is the output location of the merged file. Note that the part file contains the XML data in it and those are very huge files.

When I ran the above command, I got an error in the middle of the merge process and threw an error with “cat unable to write output stream” message.

I have decided to use getmerge command to get rid of the above error. It works fine without any issues. Check the below command.


   hadoop fs -getmerge ${INPUT_PATH}/part* > ${OUTPUT_FOLDER}/output.xml

How to Run Groovy test cases with Groovy Jar

Assume that you have a groovy file (test.groovy) and you want to run it with Groovy jar. Follow the below steps to do it.

Here is the test.groovy file

//test.groovy
println "test"

Then download the groovy.jar, If you don’t have it.

curl -s -L -o groovy.jar \
       "http://central.maven.org/maven2/org/codehaus/groovy/groovy-all/1.8.1/groovy-all-1.8.1.jar"

Then run the test file with the below command.

  java -cp groovy.jar groovy.lang.GroovyShell test.groovy

Move Jenkins jobs from old server to new server

Follow the below steps to move the Jenkins jobs from one server to an another server.

1. Download the jenkins-cli.jar from your Jenkins server(http://JENKINS.SERVER/jnlpJars/jenkins-cli.jar)

2. Then run the below command to get the job configuration and then move it to a config file. Make sure to update the SERVER NAME, JOB_NAME, USERNAME, PASSWORD accordingly


java -jar jenkins-cli.jar -s http://OLD.JENKINS.SERVER.NAME get-job JOB_NAME --username USERNAME --password PASSWORD > JOB_NAME.xml


3. Finally, run the below command to create the job on the new server.Make sure to update the SERVER NAME, JOB_NAME, USERNAME, PASSWORD accordingly


java -jar jenkins-cli.jar -s http://NEW.JENKINS.SERVER.NAME create-job JOB_NAME --
username USERNAME --password PASSWORD < JOB_NAME.xml