Tutorial – Java & Twitter Public Streams API

With support from Heroku, Spark, & OAuth

I’ve recently been using the Twitter API at work to pull social media (like Tweets) into some of the websites we are building. This got me thinking about some fun side projects and a chance to build something in Java, which I’ve been interested in. I found Twitter’s Stream API, and thought this could be a good use for a simple Java server. I’ve created this post to act as a simple tutorial which should help you get started in connecting to Twitter’s Stream API using Java and run a web server alongside it.

1. Get Your API Keys

Head over to https://dev.twitter.com to get your API keys. You’ll need to create an appication and save the following keys from the OAuth tab.
– Consumer key
– Consumer secret
– Access token
– Accesss token secret

2. Create a Java Project

Open up your favorite Java IDE and start a new project (note – we do need Maven as a build tool)

Program Structure

The Twitter Stream API allows you to connect to Twitter’s global stream of Tweet data, allowing messages to be pushed to you instead of pulling them. To use the Twitter Stream we will need to maintain a persistent connection to their server. Since we are planning to run both our own web server and maintaining a persistent connection, I decided I would create a separate class to handle Twitter stream and run this on its own thread. Now let’s create a basic class that extends Thread and can output some text to the console.

src/main/java/TwitterStreamConsumer.java

public class TwitterStreamConsumer extends Thread {

    private static final String STREAM_URI = "https://stream.twitter.com/1.1/statuses/filter.json";

    public void run(){
        System.out.println("Starting Twitter public stream consumer thread.");    
    }
}

OAuth

Nothing too exciting happened there, we can’t even run it yet… So, let’s OAuth going so we can connect to Twitter. I decided to use the OAuth Java Lib Scribe. We can get scribe using Maven by adding it to our pom.xml.

<dependency>
	<groupId>org.scribe</groupId>
	<artifactId>scribe</artifactId>
	<version>1.3.4</version>
</dependency>

### 3. Connecting to Twitter Public Stream API
Now that we’ve got our OAuth library, let’s use it to connect to twitter. We’ll need to use the keys and tokens from the Twitter developer account & application we created above.

src/main/java/TwitterStreamConsumer.java

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import org.scribe.builder.*;
import org.scribe.builder.api.*;
import org.scribe.model.*;
import org.scribe.oauth.*;

public class TwitterStreamConsumer extends Thread {

    private static final String STREAM_URI = "https://stream.twitter.com/1.1/statuses/filter.json";

    public void run(){
        try{
            System.out.println("Starting Twitter public stream consumer thread.");

            // Enter your consumer key and secret below
            OAuthService service = new ServiceBuilder()
                    .provider(TwitterApi.class)
                    .apiKey("YOUR CONSUMER KEY")
                    .apiSecret("YOUR CONSUMER SECRET")
                    .build();

            // Set your access token
            Token accessToken = new Token("YOUR OAuth ACCESS TOKEN", "YOUR OAuth ACCESS TOKEN SECRET");

            // Let's generate the request
            System.out.println("Connecting to Twitter Public Stream");
            OAuthRequest request = new OAuthRequest(Verb.POST, STREAM_URI);
            request.addHeader("version", "HTTP/1.1");
            request.addHeader("host", "stream.twitter.com");
            request.setConnectionKeepAlive(true);
            request.addHeader("user-agent", "Twitter Stream Reader");
            request.addBodyParameter("track", "java,heroku,twitter"); // Set keywords you'd like to track here
            service.signRequest(accessToken, request);
            Response response = request.send();

            // Create a reader to read Twitter's stream
            BufferedReader reader = new BufferedReader(new InputStreamReader(response.getStream()));

            String line;
            while ((line = reader.readLine()) != null) {
                System.out.println(line);
            }
        }
        catch (IOException ioe){
            ioe.printStackTrace();
        }

    }

As you can see, we begin by configuring a few details related to OAuth, create a request object (and set some headers for Twitter), send the request, and create a buffered stream reader that reads the responses stream. At this point, we’ve just about built everything we need to connect to Twitter and start outputting data from the stream. Let’s jump over to our other file which contains our main class.

src/main/java/TwitterStreamServer.java

public class TwitterStreamServer {
    public static void main(String[] args){

        final TwitterStreamConsumer streamConsumer = new TwitterStreamConsumer(); // final because we will later pull the latest Tweet
        streamConsumer.start();
    }
}

Start up your program and you should begin to see Twitter stream data coming in! (I didn’t see how much traffic the preloaded keywords get (Java, Heroku, Twitter) so if you’re not seeing anything come through try some more popular keywords)

4. Adding a web server (Spark)

While it’s great to be able to connect to the Twitter stream, it’s not very useful unless we can later do something with it. This next section will briefly cover a very simple web server that will let you display information from Twitter’s stream to on your site.

pom.xml

<repositories>
	<repository>
	  <id>Spark repository</id>
	  <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
	</repository>
</repositories>
<dependencies>
  <dependency>
    <groupId>spark</groupId>
    <artifactId>spark</artifactId>
    <version>0.9.9.4-SNAPSHOT</version>
  </dependency>
  <dependency>
    <groupId>org.scribe</groupId>
    <artifactId>scribe</artifactId>
    <version>1.3.3</version>
  </dependency>
</dependencies>

If you rebuild your program you should be able to navigate to http://localhost:4567/hello and see “Hello, spark” while your console is going nuts with the output from Twiter’s stream.

5. Putting It All Together

We’ve not got a successful connection with Twitter and a functioning web server. Let’s expand on our TwitterConsumer class to store the latest output from Twitter along with a count of messages received and make them available to our web server.

src/main/java/TwitterStreamConsumer.java

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import org.scribe.builder.*;
import org.scribe.builder.api.*;
import org.scribe.model.*;
import org.scribe.oauth.*;

import javax.print.attribute.standard.DateTimeAtCompleted;

public class TwitterStreamConsumer extends Thread {
    private static final String STREAM_URI = "https://stream.twitter.com/1.1/statuses/filter.json";
    private String latestTweet;
    private int tweetCount;

    public String getLatestTweet(){
        return latestTweet;
    }

    public int getTweetCount(){
        return tweetCount;
    }

    public void run(){
        try{
            // Enter your consumer key and secret below
            OAuthService service = new ServiceBuilder()
                    .provider(TwitterApi.class)
                    .apiKey("YOUR CONSUMER KEY")
                    .apiSecret("YOUR CONSUMER SECRET")
                    .build();

            // Set your access token
            Token accessToken = new Token("YOUR OAuth ACCESS TOKEN", "YOUR OAuth ACCESS TOKEN SECRET");

            // Let's generate the request
            System.out.println("Connecting to Twitter Public Stream");
            OAuthRequest request = new OAuthRequest(Verb.POST, STREAM_URI);
            request.addHeader("version", "HTTP/1.1");
            request.addHeader("host", "stream.twitter.com");
            request.setConnectionKeepAlive(true);
            request.addHeader("user-agent", "Twitter Stream Reader");
            request.addBodyParameter("track", "java,heroku,twitter"); // Set keywords you'd like to track here
            service.signRequest(accessToken, request);
            Response response = request.send();

            // Create a reader to read Twitter's stream
            BufferedReader reader = new BufferedReader(new InputStreamReader(response.getStream()));

            String line;
            while ((line = reader.readLine()) != null) {
                latestTweet = line;
                tweetCount++;
                //System.out.println(line);
            }
        }
        catch (IOException ioe){
            //handle the ioe
            ioe.printStackTrace();
        }

    }
}

Here you can see we’ve created a string to store the latest message (typically a tweet) received from Twitter along with a running count of the messages received. Let’s get these and output them on the web server using the two methods above.

src/main/java/TwitterStreamServer.java

import static spark.Spark.*;
import spark.*;

public class TwitterStreamServer {
    public static void main(String[] args) throws java.io.IOException{

        final TwitterStreamConsumer streamConsumer = new TwitterStreamConsumer();
        streamConsumer.start();

        //setPort(Integer.parseInt(System.getenv("PORT"))); uncomment for Heroku
        get(new Route("/hello") {
            @Override
            public Object handle(Request request, Response response) {
                return "<h1>Tweet Count: " + Integer.toString(streamConsumer.getTweetCount()) + "</h1> 
                       " + " <h2>Latest Payload: " + streamConsumer.getLatestTweet() + "</h2>";
            }
        });
    }
}

Now you should have a functioning web server which pulls data from your Twitter public stream connection.

6. Deploying on Heroku

Normally, it is pretty easy to deploy to Heroku but this app requires a small twist… Customizing the JDK, for licensing reasons the OpenJDK builds that Heroku uses don’t include the Java Cryptography Extensions (JCE), which some SSL certificates require to be processed.

Customizing the JDK for Heroku

Create a system.properties file and in your applications root directory create a .jdk-overlay folder (we’ll also create the path for our JCE files).

$ echo "java.runtime.version=1.7" > system.properties
$ mkdir -p .jdk-overlay/jre/lib/security

Next, we will need to [download the JCE] (http://www.oracle.com/technetwork/java/javase/downloads/index.html) and bring it into our .jdk-overlay

$ cp ~/downloads/jce/local_policy.jar .jdk-overlay/jre/lib/security/

Push to Heroku!

All the heavy lifting is done, uncomment a line of code and a couple git commands and we should be up and running. Let’s start by uncommenting setPort(Integer.parseInt(System.getenv("PORT"))); this will let Heroku set the port it needs to run Spark on.
Create a .gitignore file to prevent our build artifacts from getting into version control.

$ echo target > .gitignore

Next, we’ll need a Procfile (it is case sensitive!) to tell Heroku how to start the app.

Procfile

web:  java -cp target/classes:target/dependency/* YOUR_APPLICATION

Our last steps are to initialize git…

$ git init
$ git add .
$ git commit -m "init"

and deploy on Heroku!

$ heroku create
$ git push origin master

We are all done visit your Heroku apps URI + /hello to see your server in action. Since letting my app run for about a day almost 3.5 million messages from Twitter’s API have passed through my servers.