How to copy the files in between two CDH clusters

Sometimes, we may need to copy the content from one CDH5 cluster to an another CDH5 cluster. We can make use of distcp to achieve that.

Command:


hadoop distcp -m 10 -prbugpcaxt hdfs://ACTIVE_NAME_NODE_OF_CLUSTER_A:8020/FILE_PATH hdfs://ACTIVE_NAME_NODE_OF_CLUSTER_B:8020/FILE_PATH

-m stands for Maximum number of simultaneous copies. Specify the number of map operations.

-p refers Preserve r: replication number b: block size u: user g: group p: permission c: checksum-type a: ACL x: XAttr t: timestamp

Example:


hadoop distcp -m 100 -pbugp hdfs://test01-activenamenode.domain.com:8020/ops/file1 hdfs://test02-activenamenode.domain.com:8020/ops/file1

If you faced any memory issues like below during map or reduce operation, than pass the below arguments (-Dmapreduce.map.memory.mb and -Dmapreduce.reduce.memory.mb)


Container is running beyond physical memory limits. Current usage: 1.1 GB of 1 GB physical memory used; 6.1 GB of 2.1 GB virtual memory used. Killing container.
.....
......


hadoop distcp -Dmapreduce.map.memory.mb=2000 -Dmapreduce.reduce.memory.mb=2000 -m 100 -pbugp hdfs://test01-activenamenode.domain.com:8020/ops/file1 hdfs://test02-activenamenode.domain.com:8020/ops/file1

You would get an error if the file exists already. To override the destination file, then pass -update option. Refer the below command to know how to do it.


hadoop distcp -update hdfs://test01-activenamenode.domain.com:8020/mnow-ops/mnow/builds/47/_LATEST hdfs://test02-activenamenode.domain.com:8020/mnow-ops/mnow/builds/47/_LATEST


Advertisements

Java 8 Lambda – Using Map

In this post, I am going to show some examples of using Java 8 Stream map().

I have an arraylist which contains list of strings and want to change the contents of that list to upper case. We can easily do that with map() but I want to make sure that my code should work fine for all types of input such as valid list, empty list and null list.

Refer the below code to know how to do that.


import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors;

public class LambdaExample {

    public static void main(String[] args) {

        //Passing valid list
        List validList = Arrays.asList("Name1", "Name2");
        System.out.println("Converting valid list contents to upper case: " + convertAllToUpperCase(validList));
        //Passing an empty list
        System.out.println("Converting an empty list  to upper case: " + convertAllToUpperCase(Collections.emptyList()));
        //Passing null
        System.out.println("Converting null list to upper case: " + convertAllToUpperCase(null));
    }

    private static List convertAllToUpperCase(List list) {
        return Optional.ofNullable(list)
                .orElseGet(Collections::emptyList).stream().map(String::toUpperCase)
                .collect(Collectors.toList());
    }

}


Here, I have used the “Optional” class to handle null list. The code “Optional.ofNullable(list).orElseGet(Collections::emptyList)” will return an empty list when the list is null.

The output of the above program will look like below,


Converting valid list contents to upper case: [NAME1, NAME2]
Converting an empty list  to upper case: []
Converting null list to upper case: []

Now lets see an another example to show how we can map one model class to an another class via Java 8 Functional interface.

There are two model classes here. They are Person and PersonBean. So I want to iterate through the Person list and create an another list of PersonBean.

I have used Function inferface which represents a function that accepts one argument and produces a result. Here the argument is Person and the result is PersonBean.

The first step is to define the function and then use that function while mapping. Refer the below code to know how to do that.


import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Optional;
import java.util.function.Function;
import java.util.stream.Collectors;

public class LambdaExample {

    Function personBeanFunction = person -> {
        PersonBean personBean = new PersonBean();
        personBean.age = person.age;
        personBean.firstName = person.givenName;
        personBean.lastName = person.familyName;
        return personBean;
    };

    public static void main(String[] args) {
        LambdaExample lambdaExample = new LambdaExample();
        List personList = new ArrayList();
        Person person = new Person();
        person.age = 45;
        person.familyName = "test";
        person.givenName = "test";
        personList.add(person);
        System.out.println("PersonList list:" + personList);
        System.out.println("Person Bean list:" + lambdaExample.convert(personList));
    }

    private List convert(List list) {
        return Optional.ofNullable(list).orElseGet(Collections::emptyList).stream()
                .map(personBeanFunction).collect(Collectors.toList());
    }

    private static class Person {
        String familyName;
        String givenName;
        int age;

        @Override
        public String toString() {
            final StringBuilder sb = new StringBuilder("Person{");
            sb.append("familyName='").append(familyName).append('\'');
            sb.append(", givenName='").append(givenName).append('\'');
            sb.append(", age=").append(age);
            sb.append('}');
            return sb.toString();
        }
    }

    private static class PersonBean {
        String firstName;
        String lastName;
        int age;

        @Override
        public String toString() {
            final StringBuilder sb = new StringBuilder("PersonBean{");
            sb.append("firstName='").append(firstName).append('\'');
            sb.append(", lastName='").append(lastName).append('\'');
            sb.append(", age=").append(age);
            sb.append('}');
            return sb.toString();
        }
    }
}

The output of the above program will look like below,


PersonList list:[Person{familyName='test', givenName='test', age=45}]
Person Bean list:[PersonBean{firstName='test', lastName='test', age=45}]