Hadoop merge large part files with ‘getmerge’ command

I recently faced an issue while trying to merge large number of part files into a single file. I had used the ‘cat’ command for the merge. My command looks like below,

   hadoop fs -cat ${INPUT_PATH}/part* > ${OUTPUT_FOLDER}/output.xml

Assume that INPUT_PATH is the location of part files and OUTPUT_FOLDER is the output location of the merged file. Note that the part file contains the XML data in it and those are very huge files.

When I ran the above command, I got an error in the middle of the merge process and threw an error with “cat unable to write output stream” message.

I have decided to use getmerge command to get rid of the above error. It works fine without any issues. Check the below command.

   hadoop fs -getmerge ${INPUT_PATH}/part* > ${OUTPUT_FOLDER}/output.xml

How to Run Groovy test cases with Groovy Jar

Assume that you have a groovy file (test.groovy) and you want to run it with Groovy jar. Follow the below steps to do it.

Here is the test.groovy file

println "test"

Then download the groovy.jar, If you don’t have it.

curl -s -L -o groovy.jar \

Then run the test file with the below command.

  java -cp groovy.jar groovy.lang.GroovyShell test.groovy

Move Jenkins jobs from old server to new server

Follow the below steps to move the Jenkins jobs from one server to an another server.

1. Download the jenkins-cli.jar from your Jenkins server(http://JENKINS.SERVER/jnlpJars/jenkins-cli.jar)

2. Then run the below command to get the job configuration and then move it to a config file. Make sure to update the SERVER NAME, JOB_NAME, USERNAME, PASSWORD accordingly

java -jar jenkins-cli.jar -s http://OLD.JENKINS.SERVER.NAME get-job JOB_NAME --username USERNAME --password PASSWORD > JOB_NAME.xml

3. Finally, run the below command to create the job on the new server.Make sure to update the SERVER NAME, JOB_NAME, USERNAME, PASSWORD accordingly

java -jar jenkins-cli.jar -s http://NEW.JENKINS.SERVER.NAME create-job JOB_NAME --
username USERNAME --password PASSWORD < JOB_NAME.xml

Java 8: Create LinkedHashMap with Collectors.toMap

In this article, I will show you how to create a LinkdHashMap with Collectors.toMap() method.

Assume that you have a LinkedHashMap with the key as a string and the value as an Integer. Now you want to iterate this map and modify the value of each key and then create an another map to hold that value.

Refer the below code to know how to do that.

import java.util.LinkedHashMap;
import java.util.Map;
import java.util.stream.Collectors;

public class Main {

    public static void main(String[] args) {

        Map map1 = new LinkedHashMap();

        map1.put("key1", 1);
        map1.put("key2", 2);
        map1.put("key3", 3);


        Map map2 = map1.keySet()
                .collect(Collectors.toMap(key -> key,
                        key -> (map1.get(key) + 10),
                        (e1, e2) -> e1,


By default, the Collectors.toMap function creates a HashMap but we can override it by specifying the mapSupplier argument. The corresponding function from Collectors class which should be used in this case.

    public static <T, K, U, M extends Map<K, U>>
    Collector<T, ?, M> toMap(Function<? super T, ? extends K> keyMapper,
                                Function<? super T, ? extends U> valueMapper,
                                BinaryOperator<U> mergeFunction,
                                Supplier<M> mapSupplier) 

So in my code,I am passing the key, updated value, merge function(consider the first key if there is a duplicate key), finally the LinkedHashMap to the Collectors.toMap() function.

The output will be like this,

 map1{key1=1, key2=2, key3=3}
 map2{key1=11, key2=12, key3=13}

Create Docker image with Maven build

There are lots of maven Docker plugin available to integrate the docker with maven.

In this example, I am going to show how to build the Docker image while building a maven project.

Copy the below snippet and put into your pom.xml file and then create a maven property “docker.image.name” with the appropriate docker image name and also make sure that the Dockerfile available in the correct location.

Then run the ‘mvn install’ and once its done, run ‘docker images’ and check that the docker image is available in the list of images.



Create private function in AngularJs controller

If we want to create a private function inside an AngualrJS controller file, then we should prefix the function with _ symbol. Refer the below example.

class customerController {    
	// when landing on the page, get all customes and show them
	constructor($scope, $http) {
		 this.$http = $http;
		 this.customers = this._formatCustomers(this.getCustomers());
    getCustomers() {
              method: 'GET',
              url: `/api/customers`
          }).then(response => response.data);				
    _formatCustomers(data) {
       //You can put some logic here to format the customer data.           

angular.module('myApp').controller('customerController', customerController);

In the above example, customerController has two functions. They are getCustomers and _formatCustomers. Note that the first one is public and the later is private which can be accessed only within the controller js file.

‘Library Fine’ problem in Hackerrank – solution in Scala

Problem Statement:

The Head Librarian at a library wants you to make a program that calculates the fine for returning the book after the return date. You are given the actual and the expected return dates. Calculate the fine as follows:

If the book is returned on or before the expected return date, no fine will be charged, in other words fine is 0.

If the book is returned in the same month as the expected return date, Fine = 15 Hackos × Number of late days

If the book is not returned in the same month but in the same year as the expected return date, Fine = 500 Hackos × Number of late months

If the book is not returned in the same year, the fine is fixed at 10000 Hackos.

Refer the below link to know more about this problem. https://www.hackerrank.com/challenges/library-fine

Solution in Scala

import java.util.{Calendar, Scanner}

object Solution {

    def main(args: Array[String]) {
       val scanner: Scanner = new Scanner(System.in)

    //Actual Returned Date
    val aDate: Int = scanner.nextInt
    val aMonth: Int = scanner.nextInt
    val aYear: Int = scanner.nextInt

    //Due Date
    val dDate: Int = scanner.nextInt
    val dMonth: Int = scanner.nextInt
    val dYear: Int = scanner.nextInt

    val isValidData: Boolean = ((aDate >= 1 && aDate = 1 && dDate = 1 && aMonth = 1 && dMonth = 1 && aYear = 1 && dYear <= 3000))

    var fineAmount: Int = 0
    if (isValidData) {
      val actualCalendar: Calendar = Calendar.getInstance()
      actualCalendar.set(aYear, aMonth, aDate)

      val dCalendar: Calendar = Calendar.getInstance()
      dCalendar.set(dYear, dMonth, dDate)

      if ((actualCalendar.getTime == dCalendar.getTime) || actualCalendar.getTime.before(dCalendar.getTime)) {
        fineAmount = 0
      else if (actualCalendar.getTime.after(dCalendar.getTime) && aYear == dYear) {
        fineAmount = if ((aMonth == dMonth)) 15 * (aDate - dDate) else 500 * (aMonth - dMonth)
      else {
        fineAmount = 10000