Hadoop merge large part files with ‘getmerge’ command

I recently faced an issue while trying to merge large number of part files into a single file. I had used the ‘cat’ command for the merge. My command looks like below,


   hadoop fs -cat ${INPUT_PATH}/part* > ${OUTPUT_FOLDER}/output.xml

Assume that INPUT_PATH is the location of part files and OUTPUT_FOLDER is the output location of the merged file. Note that the part file contains the XML data in it and those are very huge files.

When I ran the above command, I got an error in the middle of the merge process and threw an error with “cat unable to write output stream” message.

I have decided to use getmerge command to get rid of the above error. It works fine without any issues. Check the below command.


   hadoop fs -getmerge ${INPUT_PATH}/part* > ${OUTPUT_FOLDER}/output.xml

How to Run Groovy test cases with Groovy Jar

Assume that you have a groovy file (test.groovy) and you want to run it with Groovy jar. Follow the below steps to do it.

Here is the test.groovy file

//test.groovy
println "test"

Then download the groovy.jar, If you don’t have it.

curl -s -L -o groovy.jar \
       "http://central.maven.org/maven2/org/codehaus/groovy/groovy-all/1.8.1/groovy-all-1.8.1.jar"

Then run the test file with the below command.

  java -cp groovy.jar groovy.lang.GroovyShell test.groovy

Move Jenkins jobs from old server to new server

Follow the below steps to move the Jenkins jobs from one server to an another server.

1. Download the jenkins-cli.jar from your Jenkins server(http://JENKINS.SERVER/jnlpJars/jenkins-cli.jar)

2. Then run the below command to get the job configuration and then move it to a config file. Make sure to update the SERVER NAME, JOB_NAME, USERNAME, PASSWORD accordingly


java -jar jenkins-cli.jar -s http://OLD.JENKINS.SERVER.NAME get-job JOB_NAME --username USERNAME --password PASSWORD > JOB_NAME.xml


3. Finally, run the below command to create the job on the new server.Make sure to update the SERVER NAME, JOB_NAME, USERNAME, PASSWORD accordingly


java -jar jenkins-cli.jar -s http://NEW.JENKINS.SERVER.NAME create-job JOB_NAME --
username USERNAME --password PASSWORD < JOB_NAME.xml

Java 8: Create LinkedHashMap with Collectors.toMap

In this article, I will show you how to create a LinkdHashMap with Collectors.toMap() method.

Assume that you have a LinkedHashMap with the key as a string and the value as an Integer. Now you want to iterate this map and modify the value of each key and then create an another map to hold that value.

Refer the below code to know how to do that.



import java.util.LinkedHashMap;
import java.util.Map;
import java.util.stream.Collectors;

public class Main {

    public static void main(String[] args) {

        Map map1 = new LinkedHashMap();

        map1.put("key1", 1);
        map1.put("key2", 2);
        map1.put("key3", 3);

        System.out.println(map1);

        Map map2 = map1.keySet()
                .stream()
                .collect(Collectors.toMap(key -> key,
                        key -> (map1.get(key) + 10),
                        (e1, e2) -> e1,
                        LinkedHashMap::new));

        System.out.println(map2);
    }
}


By default, the Collectors.toMap function creates a HashMap but we can override it by specifying the mapSupplier argument. The corresponding function from Collectors class which should be used in this case.


    public static <T, K, U, M extends Map<K, U>>
    Collector<T, ?, M> toMap(Function<? super T, ? extends K> keyMapper,
                                Function<? super T, ? extends U> valueMapper,
                                BinaryOperator<U> mergeFunction,
                                Supplier<M> mapSupplier) 

So in my code,I am passing the key, updated value, merge function(consider the first key if there is a duplicate key), finally the LinkedHashMap to the Collectors.toMap() function.

The output will be like this,


 map1{key1=1, key2=2, key3=3}
 map2{key1=11, key2=12, key3=13}

Create Docker image with Maven build

There are lots of maven docker plugin available to integrate the docker with maven.

In this example, I am going to show how to build the docker image while building a maven project.

Refer the code at https://github.com/dkbalachandar/helloworld-docker

pom.xml:


<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>exec-maven-plugin</artifactId>
    <version>1.4.0</version>
    <executions>
        <execution>
            <goals>
                <goal>java</goal>
            </goals>
        </execution>
        <execution>
            <id>build-image</id>
            <phase>install</phase>
            <goals>
                <goal>exec</goal>
            </goals>
            <configuration>
                <executable>docker</executable>
                <arguments>
                    <argument>build</argument>
                    <argument>-t=hello:latest</argument>
                    <argument>.</argument>
                </arguments>
            </configuration>
        </execution>
    </executions>
    <configuration>
        <mainClass>com.Grizzly</mainClass>
    </configuration>
</plugin>

Copy the above snippet and put into your pom.xml file and then update the ‘mainClasss’ accordinlgy and also make sure that the Dockerfile available in the correct location.

Then run the ‘mvn install’ and once its done, run ‘docker images’ and check that the ‘hello’ docker image is available in the list of images.

Create private function in AngularJs controller

If we want to create a private function inside an AngualrJS controller file, then we should prefix the function with _ symbol. Refer the below example.


class customerController {    
	// when landing on the page, get all customes and show them
	constructor($scope, $http) {
		 this.$http = $http;
		 this.customers = this._formatCustomers(this.getCustomers());
	}
	
    getCustomers() {
         this.$http({
              method: 'GET',
              url: `/api/customers`
          }).then(response => response.data);				
    }
	
    _formatCustomers(data) {
       //You can put some logic here to format the customer data.           
    }
}

angular.module('myApp').controller('customerController', customerController);


In the above example, customerController has two functions. They are getCustomers and _formatCustomers. Note that the first one is public and the later is private which can be accessed only within the controller js file.