HTTP Caching in REST API

In this post, I am going to show an example which explains the implementation of HTTP caching in REST API.

As we know that REST API services can be cached in the browser whereas SOAP based services are not.

We have two kinds of cache control mechanism

1. Time based cache header
2. Conditional cache header

Time based cache header

Assume that we have a web service and you want to cache the response of that service in client’s browser for 5 min. So to achieve the same, we should have to set the cache control HTTP header appropriately


Cache-Control:private, max-age=300

Here “private” denotes that the response will be cached only in the client mostly by browser and not any intermediate proxy servers

max-age denotes that how long the resource is valid. The value should be given in seconds

Conditional cache header

With the conditional cache header approach, the browser will ask the server whether the response/content has been changed or not. Here the Browser sends out ETag and If-Modified-Since headers[Need to set If-Modified-Since at the client API] to the server and at the server side we should use these headers and implement our logic and send out the response only if its changed or the client side content has been expired.

You can note for the first time, we get the HTTP status 200 and for the subsequent request, we get the HTTP status 304 Not Modified

Please refer the below sample code which shows both approaches



import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.CacheControl;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.EntityTag;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Request;
import javax.ws.rs.core.Response;

@Path("/greeter")
public class GreetingResource {

    private static final String HELLO_MESSAGE= "Welcome %s";

    private static final String BIRTHDAY_MESSAGE= "Happy Birthday %s";

    @Context
    private Request request;

    @GET
    @Path("welcome")
    @Produces(MediaType.TEXT_HTML)
    public Response sayWelcomeMessage(@QueryParam("name") String name) {
        System.out.println("sayWelcomeMessage:::::"+ System.currentTimeMillis());
        CacheControl cacheControl = new CacheControl();
        cacheControl.setMaxAge(60);
        cacheControl.setPrivate(true);
        Response.ResponseBuilder builder = Response.ok(String.format(HELLO_MESSAGE, name));
        builder.cacheControl(cacheControl);
        return builder.build();
    }

    @GET
    @Path("bday")
    @Produces(MediaType.TEXT_HTML)
    public Response sayBdayWishesMessage(@QueryParam("name") String name) {

        System.out.println("sayGreetingMessage:::::"+ System.currentTimeMillis());
        CacheControl cacheControl = new CacheControl();
        //60 seconds
        cacheControl.setMaxAge(60);

        String message = String.format(BIRTHDAY_MESSAGE, name);
        EntityTag entityTag = new EntityTag(Integer.toString(message.hashCode()));
        //Browser sends ETag. So we can use this and
        // find out whether the response is expired or not
        Response.ResponseBuilder builder = request.evaluatePreconditions(entityTag);
        // If the build is null, then the content dont match, so the response should be sent
        if (builder == null) {
            builder = Response.ok(message);
            builder.tag(entityTag);
        } else {
            builder.cacheControl(cacheControl).tag(entityTag);
        }
        return builder.build();
    }
}

Advertisements

Cascading example[Inner Join, Sub Assembly, Operation and Buffer]

In this post, I am going to show an example with Cascading API to perform the inner join on two related dataset

Assume that you have a professor and college details available in a separate XML files and you want to combine both these details and want to generate the consolidated data.

For example, here is our professor.xml file


<professor>
	<pid>PROF-100</pid>
	<pfirstname>John</pfirstname>
	<plastname>Turner</plastname>
	<college>
		<id>COL-100</id>
	</college>
</professor>

The college.xml is given below,


<college>
	<id>COL-100</id>
	<name>Ohio State University</name>
	<location>Ohio</location>
</college>

And You want to get the consolidated XML file as below,


<professor>
	<pid>PROF-100</pid>
	<pfirstname>John</pfirstname>
	<plastname>Turner</plastname>
	<college>
		<id>COL-100</id>
		<name>Ohio State University</name>
		<location>Ohio</location>
	</college>
</professor>

I have created three Sub assemblies here, ProfessorDataAssembly to extract out the Professor data, CollegeDataAssembly to extract out the College data and finally ProfessorCollegeJoinAssembly to combine both these data and generate the consolidated XML with ProfessorCollegeBuffer

Please refer the below code to know how i have done it.

ProfessorCollegeDtailsJob.java


package com.cascading;

import cascading.flow.FlowDef;
import cascading.flow.local.LocalFlowConnector;
import cascading.pipe.Pipe;
import cascading.property.AppProps;
import cascading.scheme.local.TextLine;
import cascading.tap.SinkMode;
import cascading.tap.Tap;
import cascading.tap.local.FileTap;
import cascading.tuple.Fields;
import com.cascading.assembly.ProfessorDataAssembly;
import com.cascading.assembly.ProfessorCollegeJoinAssembly;
import com.cascading.assembly.CollegeDataAssembly;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Properties;

/**
 * This cascading job is used to read Professor and College details and do inner join and
 * create a consolidated XML which has professor and college information
 */
public class ProfessorCollegeDtailsJob {

    public static Logger LOGGER = LoggerFactory.getLogger(ProfessorCollegeDtailsJob.class);

    public static void main(String[] args) {

        if (args.length <= 0) {
            LOGGER.info("Usage ProfessorCollegeDtailsJob <PROFESSOR XML FILE PATH> " +
                    "<COLLEGE XML FILE PATH> <OUTPUT FILE PATH>");
            return;
        }
        //input paths & output path
        String professorDataPath = args[0];
        String collegeDataPath = args[1];
        String outputPath = args[2];

        LOGGER.info("professorDataPath:{}", professorDataPath);
        LOGGER.info("studentDataPath:{}", collegeDataPath);
        LOGGER.info("outputPath:{}", outputPath);

        //Set the application JAR class
        Properties properties = new Properties();
        AppProps.setApplicationJarClass(properties, ProfessorCollegeDtailsJob.class);

        //Source and Sink Tap. Use Hfs for running in hadoop. if you are testing in local, then use FileTap
        Tap professorInTap = new FileTap(new TextLine(new Fields("line")), professorDataPath);
        Tap collegeInTap = new FileTap(new TextLine(new Fields("line")), collegeDataPath);
        Tap outTap = new FileTap(new TextLine(), outputPath, SinkMode.REPLACE);

        //Here goes the Pipe and assemblies
        Pipe professorPipe = new Pipe("professor");
        professorPipe = new ProfessorDataAssembly(professorPipe);
        Pipe collegePipe = new Pipe("college");
        collegePipe = new CollegeDataAssembly(collegePipe);

        Pipe profCollegePipe = new ProfessorCollegeJoinAssembly(professorPipe, collegePipe);
        //Use LocalFlowConnector if you are testing in local
        //Use HadoopFlowConnector if you are running in Hadoop
        LocalFlowConnector flowConnector = new LocalFlowConnector();
        FlowDef flowDef = FlowDef.flowDef().addSource("professor", professorInTap).
                addSource("college", collegeInTap).addTailSink(profCollegePipe, outTap).
                setName("Professor College Details Job");

        flowConnector.connect(flowDef).complete();
    }
}

CollegeDataAssembly.java


package com.cascading.assembly;

import cascading.operation.xml.XPathParser;
import cascading.pipe.Each;
import cascading.pipe.Pipe;
import cascading.pipe.SubAssembly;
import cascading.tuple.Fields;

/**
 * This class is used to extract out the college data information
 */

public class CollegeDataAssembly extends SubAssembly {

    public CollegeDataAssembly(Pipe input) {
        super(input);
        input = new Each(input, new Fields("line"),
                new XPathParser(new Fields("collegeid", "collegename", "collegelocation"),
                        "/college/id", "/college/name", "/college/location"));
        setTails(input);
    }
}

ProfessorDataAssembly.java


package com.cascading.assembly;

import cascading.operation.xml.XPathParser;
import cascading.pipe.Each;
import cascading.pipe.Pipe;
import cascading.pipe.SubAssembly;
import cascading.tuple.Fields;

/**
 * This class is used to extract out the professor data information
 */

public class ProfessorDataAssembly extends SubAssembly {

    private static Fields fields = new Fields("pid", "pfirstname", "plastname", "profcollegeid");

    public ProfessorDataAssembly(Pipe input) {
        super(input);
        input = new Each(input, new Fields("line"),
                new XPathParser(fields,
                        "/professor/pid",
                        "/professor/pfirstname",
                        "/professor/plastname",
                        "/professor/college/id"));
        setTails(input);
    }
}


ProfessorCollegeJoinAssembly.java


package com.cascading.assembly;

import cascading.flow.FlowProcess;
import cascading.operation.BaseOperation;
import cascading.operation.Function;
import cascading.operation.FunctionCall;
import cascading.pipe.CoGroup;
import cascading.pipe.Each;
import cascading.pipe.Every;
import cascading.pipe.Pipe;
import cascading.pipe.SubAssembly;
import cascading.pipe.joiner.InnerJoin;
import cascading.tuple.Fields;
import cascading.tuple.TupleEntry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Iterator;

/**
 * This will do the inner job on professor and college data and create a consolidated XML
 *
 */
public class ProfessorCollegeJoinAssembly extends SubAssembly {

    public static Logger LOGGER = LoggerFactory.getLogger(ProfessorCollegeJoinAssembly.class);

    Fields fields = new Fields("professor");

    public ProfessorCollegeJoinAssembly(Pipe professorPipe, Pipe collegePipe) {
        super(professorPipe, collegePipe);
        professorPipe = new Each(professorPipe, new PrintDataFunction(Fields.ALL));
        collegePipe = new Each(collegePipe, new PrintDataFunction(Fields.ALL));
        Pipe profCollegePipe = new CoGroup("profCollegePipe",
                professorPipe,
                new Fields("profcollegeid"), collegePipe,
                new Fields("collegeid"),
                new InnerJoin());
        profCollegePipe = new Every(profCollegePipe, new ProfessorCollegeBuffer(new Fields("professor")), fields);
        setTails(profCollegePipe);
    }

    public static class PrintDataFunction extends BaseOperation implements Function {

        private static final long serialVersionUID = -5108505951262118306L;

        public PrintDataFunction(Fields fields) {
            super(1, fields);
        }

        @Override
        public void operate(FlowProcess flowProcess, FunctionCall functionCall) {
            TupleEntry arguments = functionCall.getArguments();
            if (arguments == null || arguments.getString(0) == null) {
                return;
            }
            Iterator itr = arguments.getTuple().iterator();
            while (itr.hasNext()) {
                LOGGER.info((String) itr.next());
            }
            functionCall.getOutputCollector().add(arguments.getTuple());
        }
    }
}

ProfessorCollegeBuffer.java


package com.cascading.assembly;

import cascading.flow.FlowProcess;
import cascading.operation.BaseOperation;
import cascading.operation.Buffer;
import cascading.operation.BufferCall;
import cascading.operation.OperationCall;
import cascading.tuple.Fields;
import cascading.tuple.Tuple;
import cascading.tuple.TupleEntry;

import java.util.Iterator;

/**
 * Create the consolidated output xml
 */

public class ProfessorCollegeBuffer extends BaseOperation implements Buffer {

    public ProfessorCollegeBuffer(Fields outputFieldName) {
        super(outputFieldName);
    }

    @Override
    public void prepare(FlowProcess flowProcess, OperationCall operationCall) {
        BufferCall bufferCall = (BufferCall) operationCall;
        bufferCall.setRetainValues(true);
    }

    @Override
    public void operate(FlowProcess flowProcess, BufferCall bufferCall) {
        Iterator iter = bufferCall.getArgumentsIterator();
        Tuple innerTuple = new Tuple();

        while (iter.hasNext()) {
            TupleEntry entry = iter.next();
            Tuple output = new Tuple();
            output.add("<professor>");
            output.add(entry.getString("pid"));
            output.add(entry.getString("pfirstname"));
            output.add(entry.getString("plastname"));
            output.add("<college>");
            output.add(entry.getString("collegeid"));
            output.add(entry.getString("collegename"));
            output.add(entry.getString("collegelocation"));
            output.add("</college>");
            output.add("</professor>");
            innerTuple.add(output);
        }
        Tuple outputTuple = new Tuple();
        outputTuple.add(innerTuple);
        bufferCall.getOutputCollector().add(outputTuple);
    }
}

Cascading job to count the empty tags in a XML file

The below cascading job is used to count the empty tags in a XML file and exports the output to a text file.



package com.cascading;

import cascading.flow.FlowDef;
import cascading.flow.FlowProcess;
import cascading.flow.hadoop.HadoopFlowConnector;
import cascading.operation.BaseOperation;
import cascading.operation.Function;
import cascading.operation.FunctionCall;
import cascading.pipe.Each;
import cascading.pipe.Pipe;
import cascading.property.AppProps;
import cascading.scheme.hadoop.TextDelimited;
import cascading.scheme.hadoop.TextLine;
import cascading.tap.SinkMode;
import cascading.tap.Tap;
import cascading.tap.hadoop.Hfs;
import cascading.tuple.Fields;
import cascading.tuple.Tuple;
import cascading.tuple.TupleEntry;
import org.apache.commons.lang3.StringUtils;
import org.jdom2.JDOMException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;

//import cascading.scheme.local.TextDelimited;
//import cascading.scheme.local.TextLine;

/**
 * Cascading job to count the number of empty tags found in a XML file.
 */

public class CascadingEmptyTagCounter {

    public static Logger LOGGER = LoggerFactory.getLogger(CascadingEmptyTagCounter.class);

    public static void main(String[] args) {

        if (args.length <= 0) {
            LOGGER.info("Usage CascadingEmptyTagCounter <INPUT> <OUTPUT> ");
            return;
        }

        //input path & output path
        String inputPath = args[0];
        String outputPath = args[1];

        LOGGER.info("inputPath:{}", inputPath);
        LOGGER.info("outputPath:{}", outputPath);

        //Set the application JAR class
        Properties properties = new Properties();
        AppProps.setApplicationJarClass(properties, CascadingEmptyTagCounter.class);

        //Source and Sink Tap. Use Hfs. if you are testing in local, then use FileTap
        Tap inTap = new Hfs(new TextLine(new Fields("line")), inputPath);
        Tap outTap = new Hfs(new TextDelimited(new Fields("line")), outputPath, SinkMode.REPLACE);

        Pipe input = new Pipe("input");
        Pipe dataPipe = new Each(input, new EmptyTagCounter(Fields.ALL));

        //Use LocalFlowConnector if you are testing in local
        HadoopFlowConnector flowConnector = new HadoopFlowConnector();
        FlowDef flowDef = FlowDef.flowDef().addSource(dataPipe, inTap).addTailSink(dataPipe, outTap).
                setName("Cascading extractor Job");
        flowConnector.connect(flowDef).complete();
    }

    /**
     * Cascading function to count the Empty tags
     */
    public static class EmptyTagCounter extends BaseOperation implements Function {

        private static final long serialVersionUID = 1L;

        private static List tags = Arrays.asList("<sub />", "<sup />", "<b />", "<i />");

        public EmptyTagCounter(Fields fields) {
            super(1, fields);
        }

        @Override
        public void operate(FlowProcess flowProcess, FunctionCall functionCall) {
            TupleEntry arguments = functionCall.getArguments();
            if (arguments == null || arguments.getString(0) == null) {
                return;
            }
            Tuple tuple = new Tuple();
            try {
                Map tagCountMap = getEmptyTagCounts(arguments.getTuple().getString(0));
                Set tagsSet = tagCountMap.entrySet();
                StringBuilder tagCounter = new StringBuilder();
                for (Map.Entry tagEntry : tagsSet) {
                    tagCounter.append(tagEntry.getKey() + "::" + tagEntry.getValue()).append("\n");
                }
                tuple.add(tagCounter.toString());
            } catch (JDOMException | IOException e) {
                LOGGER.error("XML parsing error", e);
            }
            functionCall.getOutputCollector().add(tuple);
        }

        public static Map getEmptyTagCounts(String xmlData) throws JDOMException, IOException {
            Map tagsCountMap = new HashMap();
            for (String tag : tags) {
                tagsCountMap.put(tag, String.valueOf(StringUtils.countMatches(xmlData, tag)));
            }
            return tagsCountMap;
        }
    }
}


Cascading Job to remove Empty Tags from a XML file

The below cascading job is used to remove the empty tags found in a XML file.


package com.cascading;

import cascading.flow.FlowDef;
import cascading.flow.FlowProcess;
import cascading.flow.hadoop.HadoopFlowConnector;
import cascading.operation.BaseOperation;
import cascading.operation.Function;
import cascading.operation.FunctionCall;
import cascading.pipe.Each;
import cascading.pipe.Pipe;
import cascading.property.AppProps;
import cascading.scheme.hadoop.TextDelimited;
import cascading.scheme.hadoop.TextLine;
import cascading.tap.SinkMode;
import cascading.tap.Tap;
import cascading.tap.hadoop.Hfs;
import cascading.tuple.Fields;
import cascading.tuple.Tuple;
import cascading.tuple.TupleEntry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Arrays;
import java.util.List;
import java.util.Properties;

//import cascading.flow.local.LocalFlowConnector;
//import cascading.scheme.local.TextDelimited;
//import cascading.scheme.local.TextLine;
//import cascading.tap.local.FileTap;

/**
 * Cascading job to replace the empty html tags found in a XML file
 */

public class CascadingEmptyTagReplacer {

    public static Logger LOGGER = LoggerFactory.getLogger(CascadingEmptyTagReplacer.class);

    public static void main(String[] args) {

        if (args.length <= 0) {
            LOGGER.info("Usage CascadingEmptyTagReplacer <INPUT> <OUTPUT>");
            return;
        }
        //input path & output path
        String inputPath = args[0];
        String outputPath = args[1];

        LOGGER.info("inputPath:{}", inputPath);
        LOGGER.info("outputPath:{}", outputPath);

        //Set the application JAR class
        Properties properties = new Properties();
        AppProps.setApplicationJarClass(properties, CascadingEmptyTagReplacer.class);

        //Source and Sink Tap. Use Hfs. if you are testing in local, then use FileTap
        Tap inTap = new Hfs(new TextLine(new Fields("line")), inputPath);
        Tap outTap = new Hfs(new TextDelimited(new Fields("line")), outputPath, SinkMode.REPLACE);

        Pipe input = new Pipe("input");
        Pipe dataPipe = new Each(input, new EmptyTagReplacer(Fields.ALL));

        //Use LocalFlowConnector if you are testing local
        HadoopFlowConnector flowConnector = new HadoopFlowConnector();
        FlowDef flowDef = FlowDef.flowDef().addSource(dataPipe, inTap).addTailSink(dataPipe, outTap).
                setName("Cascading Empty Tag Replacer Job");
        flowConnector.connect(flowDef).complete();
    }

    /**
     * Custom Function to replace Empty tags in the XML content
     */
    public static class EmptyTagReplacer extends BaseOperation implements Function {

        private static final long serialVersionUID = -5108505951262118306L;

        private static List tags = Arrays.asList("<sub/>", "<sup/>", "<b/>", "<i/>");

        public EmptyTagReplacer(Fields fields) {
            super(1, fields);
        }

        @Override
        public void operate(FlowProcess flowProcess, FunctionCall functionCall) {
            TupleEntry arguments = functionCall.getArguments();
            if (arguments == null || arguments.getString(0) == null) {
                return;
            }
            Tuple tuple = new Tuple();
            String xmlData = arguments.getTuple().getString(0);
            for (String tag : tags) {
                xmlData = xmlData.replace(tag, "");
            }
            tuple.add(xmlData);
            functionCall.getOutputCollector().add(tuple);
        }
    }
}


GSON library ignores new/missing fields

I have recently came across GSON library and found out that it ignores new/missing fields in the JSON response while doing unmarshalling by default. We won’t need to specify any annotations or properties to do so.

Please refer the below example

pom.xml


 <dependency>
    <groupId>com.google.code.gson</groupId>
    <artifactId>gson</artifactId>
    <version>2.7</version>
</dependency>

User.java


import org.apache.commons.lang3.builder.EqualsBuilder;
import org.apache.commons.lang3.builder.HashCodeBuilder;
import org.apache.commons.lang3.builder.ToStringBuilder;

import java.io.Serializable;

public class User implements Serializable {

    private static final long serialVersionUID = -8486882160753981372L;

    private String fname;

    private String lastname;

    public String getFname() {
        return fname;
    }

    public void setFname(String fname) {
        this.fname = fname;
    }

    public String getLastname() {
        return lastname;
    }

    public void setLastname(String lastname) {
        this.lastname = lastname;
    }

    @Override
    public String toString() {
        return new ToStringBuilder(this)
                .append("fname", fname)
                .append("lastname", lastname)
                .toString();
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;

        if (o == null || getClass() != o.getClass()) return false;

        User user = (User) o;

        return new EqualsBuilder()
                .append(fname, user.fname)
                .append(lastname, user.lastname)
                .isEquals();
    }

    @Override
    public int hashCode() {
        return new HashCodeBuilder(17, 37)
                .append(fname)
                .append(lastname)
                .toHashCode();
    }
}

GsonMain.java


import com.google.gson.Gson;

public class GsonMain {

    private static Gson gson = new Gson();

    public static void main(String[] args) {
        String response = "{\"fname\": \"bala,\", \"mname\": \"k,\", \"lastname\": \"samy\"}";
        User user = (User) fromJson(response, User.class);
        System.out.println("Converting JSON to object. user:" + user);
        System.out.println("Converting Object to JSON. user:" + toJson(user));
    }

    public static Object fromJson(String response, Class className) {
        return gson.fromJson(response, className);

    }

    public static String toJson(User user) {
        return gson.toJson(user);

    }
}

In GsonMain.main(), I try to parse the response to User object and vice versa. In User class, I have only fname and lastname but in the response string has a new field and its value[mname]. So while parsing the response, we wont see any error since the GSON lib ignores that field and parse only the known fields.

How to do Passwordless SSH

I have recently faced a scenario where it requires me to do passwordless SSH on a remote machine and again do the passwordless SSH on the same remote machine.

Here is the scenario,

My VM ==> Remote Machine==> Then do passwordless ssh on Remote Machine

As a first part, I have configured the Passwordless SSH on Remote machine by following the below steps,

1. Run ssh-keygen -t dsa -f ~/.ssh/id_dsa to generate public and private key pair
2. Run ssh-copy-id USERNAME@REMOTE_MACHINE to copy the keys to remote machine
3. The login into REMOTE_MACHINE and go to /home/USERNAME/.ssh and check the authorized_keys file and make sure that the key is there with REMOTE_MACHINE.local
4. Then run ssh USERNAME@REMOTE_MACHINE to verify it

This works fine. But once I login into REMOTE_MACHINE when i try to run ‘ssh USERNAME@REMOTE_MACHINE’. Its asking me the password again. So to enable the password less SSH, I have followed the below steps

1. Login into REMOTE_MACHINE
2. Then generate the keys ssh-keygen -t dsa -P ” -f ~/.ssh/id_dsa
3. Then run this cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

npm install – Cannot find module ‘semver’ issue

I have recently got an error “Cannot find module ‘semver'” on my Ubuntu VM when doing npm install.

+ npm install
...
...
module.js:340
    throw err;
          ^
Error: Cannot find module 'semver'

 

I have tried various things but nothing worked out. Then I have done the below things and it resolved my issue.

sudo apt-get remove nodejs

sudo apt-get remove npm

sudo apt-get update 

sudo apt-get install nodejs

sudo apt-get install npm

sudo ln -s /usr/bin/nodejs /usr/bin/node

npm install