How to use Maven Shade Plugin

Maven Shade Plugin provides the capability to package the artifact in an uber-jar, including its dependencies and to shade(rename) the packages of some of the dependencies.

This plugin is used to take all the dependencies and extract the content and put them as is and create one big JAR(Uber. This will be good for runnable jar but if you are planning to use this jar as a maven dependency then there may be some conflicts with the client jar. So we need to use the relocations option to relocate the conflicting classes to a different package.

In this post, I will show how to use this plugin with an example.

The below example is for creating the final Uber jar without relocations.

Create Uber Jar without relocations

Consider that we have a web app which exposes a REST web service and running on a grizzly container. Now, let’s use the shade plugin to create the fat jar.

I am not going to give the code examples here as this post explains about the shade plugin only.

Refer the complete code https://github.com/dkbalachandar/greeting-rest

The below plugin configuration has to be added in the maven pom file.

Refer the configuration section where we have to specify various options. They are given below,

• mainclass – The entry point of the application
• createSourcesJar – Create the sources jar as well
• shadedClassifierName – The name of the classifier used in case the shaded artifact is attached
• shadedArtifactAttached – It defines whether the shaded artifact should be attached as classifier to the original artifact. If its false, then the original jar will be replaced by the shaded jar

    <plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-shade-plugin</artifactId>
	<version>3.1.0</version>
	<executions>
		<execution>
			<phase>package</phase>
			<goals>
				<goal>shade</goal>
			</goals>
			<configuration>
				<createSourcesJar>true</createSourcesJar>
				<shadedClassifierName>shaded</shadedClassifierName>
				<shadedArtifactAttached>true</shadedArtifactAttached>
				<transformers>
					<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
						<mainClass>com.Grizzly</mainClass>
					</transformer>
				</transformers>
				<filters>
					<filter>
						<artifact>*:*</artifact>
						<excludes>
							<exclude>META-INF/*.SF</exclude>
							<exclude>META-INF/*.DSA</exclude>
							<exclude>META-INF/*.RSA</exclude>
						</excludes>
					</filter>
				</filters>                          
			</configuration>
		</execution>
	</executions>
</plugin>

Then execute ‘mvn package’ command and then check the target folder. You will be able to see the shaded jar and it ends with shaded suffix. Refer the below screenshot.

When you extract out the contents of the shaded jar, you will see below.

Create Uber Jar with relocations

Now we are going to see how we can relocate some of the classes inside the Uber Jar. Lets use the same example and add the below relocations inside the configuration element.

  <configuration>
	<createSourcesJar>true</createSourcesJar>
	<shadedClassifierName>shaded</shadedClassifierName>
	<shadedArtifactAttached>true</shadedArtifactAttached>
	<transformers>
		<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
			<mainClass>com.Grizzly</mainClass>
		</transformer>
	</transformers>
	<filters>
		<filter>
			<artifact>*:*</artifact>
			<excludes>
				<exclude>META-INF/*.SF</exclude>
				<exclude>META-INF/*.DSA</exclude>
				<exclude>META-INF/*.RSA</exclude>
			</excludes>
		</filter>
	</filters>
	<relocations>
		<relocation>
			<pattern>org.glassfish.grizzly.utils</pattern>
			<shadedPattern>org.shaded.glassfish.grizzly.utils</shadedPattern>
			<excludes>
				<exclude>org.glassfish.grizzly.utils.conditions.*</exclude>
			</excludes>
		</relocation>
	</relocations>
</configuration>

Check the element and the child elements. Here, I am planning to rename/move the org.glassfish.grizzly.utils package to org.shaded.glassfish.grizzly.utils and exclude the package org.glassfish.grizzly.utils.conditions.

Then run the mvn package again to create the shaded jar and extract out the contents of the shaded jar. It will look like below.

As you can see that the package org.glassfish.grizzly.utils has been moved to org.shaded.glassfish.grizzly.utils but the inner package org.glassfish.grizzly.utils.conditions has been kept it as in the original location.

While relocating the packages, this plugin will also update the affected bytecode. So you will not see any issues while running the Uber jar.

Turn off doclint in JDK 8

Java 8 introduced DocLint which is a quite strict linter for javadoc. By default, the build will be failed if there are any errors while generating the javadocs.

We can disable this check. Refer below to know how to do that.

If you are using the older version of maven-javadoc-plugin (2.10.2 or older). then add the “-Xdoclint:none” inside of the configuration element. Refer below.

<plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-javadoc-plugin</artifactId>
      <version>2.10.2</version>
      <configuration>
         <additionalparam>-Xdoclint:none</additionalparam>
      </configuration>
  </plugin>   

If you are using the latest version of maven-javadoc-plugin(3.0.0), then add “none” inside of the configuration element. Refer below.

<plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-javadoc-plugin</artifactId>
      <version>3.0.0</version>
      <configuration>
         <doclint>none</doclint>
      </configuration>
  </plugin>

If you would not like to add the entire plugin details, Then you can add these configurations in the global maven properties section. Like below.

<properties>
   <additionalparam>-Xdoclint:none</additionalparam>
</properties>

or

<properties>
   <doclint>none</doclint>
</properties>

Create Docker image with Maven build

There are lots of maven Docker plugin available to integrate the docker with maven.

In this example, I am going to show how to build the Docker image while building a maven project.

Copy the below snippet and put into your pom.xml file and then create a maven property “docker.image.name” with the appropriate docker image name and also make sure that the Dockerfile available in the correct location.

Then run the ‘mvn install’ and once its done, run ‘docker images’ and check that the docker image is available in the list of images.

pom.xml:

  <plugin>
	<groupId>org.codehaus.mojo</groupId>		
	<artifactId>exec-maven-plugin</artifactId>		
	<version>1.4.0</version>		
	<executions>		
		<execution>		
			<goals>		
				<goal>java</goal>		
			</goals>		
		</execution>		
		<execution>		
			<id>build-image</id>		
			<phase>install</phase>		
			<goals>		
				<goal>exec</goal>		
			</goals>		
			<configuration>		
				<executable>docker</executable>		
				<arguments>		
					<argument>build</argument>		
					<argument>-t=${docker.image.name}</argument>		
					<argument>.</argument>		
				</arguments>		
			</configuration>		
		</execution>		
	</executions>			
 </plugin>		
<plugin>
      

Spark Scala Unit Testing

In this post, I am going to show an example for writing unit test cases for Spark Scala job and run it with Maven.

Assume that we have a set of XML files which has user information like first name, last name and etc. Assume that middle name and county name are optional fields but the XML file does contain empty nodes for these two fields. So now our job is to read those files and remove those empty nodes and output those updated content into a text file either in local env or hadoop env.

The sample XML content is given below,

 

<persons>
    <person>
        <firstName>Bala</firstName>
        <middleName/>
        <lastName>Samy</lastName>
        <countyName/>
    </person>
    <person>
        <firstName>Bala1</firstName>
        <middleName/>
        <lastName>Samy1</lastName>
        <countyName/>
    </person>
</persons>

 

The Spark scala code for reading XML files and removing the empty nodes are given below.

package com

import org.apache.spark.{SparkConf, SparkContext}

import scala.collection.Map

object EmptyTagReplacer {

  def main(args: Array[String]) {

    if (args.length < 2) {
      println("Usage <inputDir> <outputDir>")
    }
    val conf = new SparkConf().setAppName("EmptyTagReplacer")
    val sc = new SparkContext(conf)

    val inFile = args(0)
    val outFile = args(1)

    val input: Map[String, String] = sc.wholeTextFiles(inFile).collectAsMap()
    searchAndReplaceEmptyTags(sc, input, outFile)
    sc.stop()
  }

  def searchAndReplaceEmptyTags(sc: SparkContext, inputXml: Map[String, String], outFile: String):
  scala.collection.mutable.ListBuffer[String] = {

    var outputXml = new scala.collection.mutable.ListBuffer[String]()
    val htmlTags = List("<middleName/>", "<countyName/>")
    inputXml.foreach { case (fileName, content) =>
      var newContent = content
      for (tag  <- htmlTags) {
        val data = sc.parallelize(newContent)
        data.saveAsTextFile(outFile + "/" + fileName)
      }
      outputXml += newContent
    }
    outputXml
  }

  def countTags(sc: SparkContext, xmlRecords: List[String]): List[Int] = {

    var middleNameTagCounter = sc.accumulator(0)
    var countyTagCounter = sc.accumulator(0)
    val middleNameRegex = "<middleName/>".r
    val countyRegEx = "<countyName/>".r
    xmlRecords.foreach { content =>
      middleNameTagCounter += middleNameRegex.findAllIn(content).length
      countyTagCounter += countyRegEx.findAllIn(content).length
    }
    List(middleNameTagCounter.value, countyTagCounter.value)
  }
}

Now the test case for testing the above spark job is given below,

package com

import java.io.File

import com.holdenkarau.spark.testing.SharedSparkContext
import org.apache.commons.io.FileUtils
import org.scalatest.FunSuite
import collection.mutable.Map

//import scala.io.Source._

class EmptyTagReplacerTest extends FunSuite with SharedSparkContext {


  test("Empty HTML tag replacer test") {

    //Read the content and create a content Map.
    //val content: String = scala.io.Source.fromFile("./src/test/resources/text-files/xml1").mkString
    val content: String =  FileUtils.readFileToString(new File("./src/test/resources/text-files/xml1"), "UTF-8")

    println("content"+content)
    val contentMap = collection.mutable.Map[String, String]()
    contentMap.+=("fileName" -> content)
    //Call searchAndReplaceMethod to remove empty Nodes
    val outputContent: scala.collection.mutable.ListBuffer[String] = EmptyTagReplacer.searchAndReplaceEmptyTags(sc, contentMap, "")
    val counts: List[Int] = EmptyTagReplacer.countTags(sc, outputContent.toList)
    println(counts)
    val expected = List(0, 0)
    assert(counts == expected)
  }
}

You have to include the scala-maven-plugin and scalatest-maven-plugin in pom.xml to make this work.

Please refer my github repo to know more https://github.com/dkbalachandar/scala-spark-test

How to integrate JaCoCo Code coverage tool with Maven

I have used Cobertura code coverage tool for one of my recent project and followed the steps mentioned in this link Cobertura Example.

When I have tried to upgrade Java version to 1.8, got some issues due to the usage of Lambda expression. So I am looking for an another code coverage tool. So I choose  JaCoCo

Assume that I have a Maven Java project and some unit test cases. I have given the below snippet from my Maven pom.xml below. Just copy and paste this into the build section and then run your build. After the build is done, then go to below location (target/site/jacoco-ut/) and click on index.html file to view the code coverage report

<plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.7.7.201606060606</version>
                <configuration>
                 <!--  Add it here to exclude it from code coverage analysis -> 
                    <excludes>
                        <exclude>**/model/**</exclude>
                        <exclude>**/test/**</exclude>                        
                    </excludes>
                </configuration>
                <executions>
                    <execution>
                        <id>pre-unit-test</id>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                        <configuration>
                            <destFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</destFile>
                            <propertyName>surefireArgLine</propertyName>
                        </configuration>
                    </execution>
                    <execution>
                        <id>post-unit-test</id>
                        <phase>test</phase>
                        <goals>
                            <goal>report</goal>
                        </goals>
                        <configuration>
                            <dataFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</dataFile>
                            <outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
                        </configuration>
                    </execution>
                    <execution>
                        <id>pre-integration-test</id>
                        <phase>pre-integration-test</phase>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                        <configuration>
                            <destFile>${project.build.directory}/coverage-reports/jacoco-it.exec</destFile>
                            <propertyName>failsafeArgLine</propertyName>
                        </configuration>
                    </execution>
                    <execution>
                        <id>post-integration-test</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>report</goal>
                        </goals>
                        <configuration>
                            <dataFile>${project.build.directory}/coverage-reports/jacoco-it.exec</dataFile>
                            <outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.19.1</version>
                <configuration>
                    <argLine>${surefireArgLine}</argLine>
                    <excludes>
                        <exclude>**/*IntegrationTest*</exclude>
                    </excludes>
                </configuration>
            </plugin>

Java 8 + Cobertura maven plugin – net/sourceforge/cobertura/coveragedata/TouchCollector – No class def found error

I have got the below stack trace while upgrading the java version from 1.7 to 1.8 and using cobertura maven plugin version 2.7.

 

java.lang.NoClassDefFoundError: net/sourceforge/cobertura/coveragedata/TouchCollector
    at [org.package.ClassName].__cobertura_init([ClassName].java)
    at [org.package.ClassName].([ClassName].java)
    at [org.package.ClassName]Test.[method]([ClassName]Test.java:113)
Caused by: java.lang.ClassNotFoundException: net.sourceforge.cobertura.coveragedata.TouchCollector

 

I have tried with various options but nothing worked out. But solved it by following the workaround given in the below link

http://www.befreeman.com/2014/09/getting-cobertura-code-coverage-with.html

If you have used Lamda expression, then its better to use any other code coverage tool other than Cobertura

Refer my another post to know how to use Jacoco code coverage tool How to integrate JaCoCo Code coverage tool with Maven

Copy File/Directory with Maven resource plugin

We can use Maven resource plugin to copy files and directory to any folder/path. Please refer the below snippet from pom.xml. This will be used to copy the config files from ${project.build.directory}/config to /opt/config folder.

 <plugin>
        <artifactId>maven-resources-plugin</artifactId>
        <version>2.7</version>
        <executions>
          <execution>
            <id>copy-resources</id>
            <phase>package</phase>
            <goals>
              <goal>copy-resources</goal>
            </goals>
            <configuration>
              <outputDirectory>/opt/config</outputDirectory>
              <resources>
                <resource>
                  <directory>${project.build.directory}/config</directory>
                  <filtering>false</filtering>
                </resource>
              </resources>
            </configuration>
          </execution>         
        </executions>
      </plugin>

Create a Fat JAR

To create a FAT jar contains all the dependent classes and Jars, Please use the below approach

Maven:

 

SBT:

Add the sbt-assembly plugin in the plugins.sbt as below

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.1")

Change the version appropriately and run sbt assembly to create the Fat Jar

mvn install vs mvn package

mvn install:

It will compile the code and also package it and finally install the jar in the local repository so that other projects can use it

mvn package:

It will also compile the code and package it but it won’t install the packaged jar in the local repository

MAVEN – Reduce build time in local enviornment

In most of the Java projects, Maven is being used as a build tool. We use “mvn –U install” command to build the project. While building the project, the maven will download the dependent jars (mentioned in POM.xml file) from the project specific repository/maven repository. Once the build is completed, then all the downloaded jars will be stored in local machine(C:/Users/<>/.m2).

Please run .m2 command on the Run window and check it out.

This kind of build process will take more time as it needs to download all the jars. But this is not required all the times. We can use the downloaded local jars available in the local repository for building the project in the next time

To do that we need to pass –o option while building our project. For example, mvn –o install. This will speed up our code build time.