Tutorial – Java & Twitter Public Streams API
With support from Heroku, Spark, & OAuth
I’ve recently been using the Twitter API at work to pull social media (like Tweets) into some of the websites we are building. This got me thinking about some fun side projects and a chance to build something in Java, which I’ve been interested in. I found Twitter’s Stream API, and thought this could be a good use for a simple Java server. I’ve created this post to act as a simple tutorial which should help you get started in connecting to Twitter’s Stream API using Java and run a web server alongside it.
1. Get Your API Keys
Head over to https://dev.twitter.com to get your API keys. You’ll need to create an appication and save the following keys from the OAuth tab.
– Consumer key
– Consumer secret
– Access token
– Accesss token secret
2. Create a Java Project
Open up your favorite Java IDE and start a new project (note – we do need Maven as a build tool)
The Twitter Stream API allows you to connect to Twitter’s global stream of Tweet data, allowing messages to be pushed to you instead of pulling them. To use the Twitter Stream we will need to maintain a persistent connection to their server. Since we are planning to run both our own web server and maintaining a persistent connection, I decided I would create a separate class to handle Twitter stream and run this on its own thread. Now let’s create a basic class that extends Thread and can output some text to the console.
Nothing too exciting happened there, we can’t even run it yet… So, let’s OAuth going so we can connect to Twitter. I decided to use the OAuth Java Lib Scribe. We can get scribe using Maven by adding it to our pom.xml.
### 3. Connecting to Twitter Public Stream API
Now that we’ve got our OAuth library, let’s use it to connect to twitter. We’ll need to use the keys and tokens from the Twitter developer account & application we created above.
As you can see, we begin by configuring a few details related to OAuth, create a request object (and set some headers for Twitter), send the request, and create a buffered stream reader that reads the responses stream. At this point, we’ve just about built everything we need to connect to Twitter and start outputting data from the stream. Let’s jump over to our other file which contains our main class.
Start up your program and you should begin to see Twitter stream data coming in! (I didn’t see how much traffic the preloaded keywords get (Java, Heroku, Twitter) so if you’re not seeing anything come through try some more popular keywords)
4. Adding a web server (Spark)
While it’s great to be able to connect to the Twitter stream, it’s not very useful unless we can later do something with it. This next section will briefly cover a very simple web server that will let you display information from Twitter’s stream to on your site.
If you rebuild your program you should be able to navigate to http://localhost:4567/hello and see “Hello, spark” while your console is going nuts with the output from Twiter’s stream.
5. Putting It All Together
We’ve not got a successful connection with Twitter and a functioning web server. Let’s expand on our TwitterConsumer class to store the latest output from Twitter along with a count of messages received and make them available to our web server.
Here you can see we’ve created a string to store the latest message (typically a tweet) received from Twitter along with a running count of the messages received. Let’s get these and output them on the web server using the two methods above.
Now you should have a functioning web server which pulls data from your Twitter public stream connection.
6. Deploying on Heroku
Normally, it is pretty easy to deploy to Heroku but this app requires a small twist… Customizing the JDK, for licensing reasons the OpenJDK builds that Heroku uses don’t include the Java Cryptography Extensions (JCE), which some SSL certificates require to be processed.
Customizing the JDK for Heroku
system.properties file and in your applications root directory create a
.jdk-overlay folder (we’ll also create the path for our JCE files).
Next, we will need to [download the JCE] (http://www.oracle.com/technetwork/java/javase/downloads/index.html) and bring it into our .jdk-overlay
Push to Heroku!
All the heavy lifting is done, uncomment a line of code and a couple git commands and we should be up and running. Let’s start by uncommenting
setPort(Integer.parseInt(System.getenv("PORT"))); this will let Heroku set the port it needs to run Spark on.
.gitignore file to prevent our build artifacts from getting into version control.
Next, we’ll need a
Procfile (it is case sensitive!) to tell Heroku how to start the app.
web: java -cp target/classes:target/dependency/* YOUR_APPLICATION
Our last steps are to initialize git…
and deploy on Heroku!
We are all done visit your Heroku apps URI + /hello to see your server in action. Since letting my app run for about a day almost 3.5 million messages from Twitter’s API have passed through my servers.