Oracle Stream Analytics platform provides a compelling combination of an easy-to-use visual façade to rapidly create and dynamically change Real Time Event Stream Processing (Fast Data) applications.
This blog shows an example on how to Integrate Twitter with Oracle Stream Analytics for realtime tweet analysis.
Use Case
Here I have taken an usecase of sentiment analysis. We will capture realtime tweets on recently released bollywood movie “Sarkar 3” and identify sentiments based on keywords defined in datasource. Please note intent of this blog is not to provide sentiment analysis solution. As there are more complex system and analytics required for it. However this blog gives idea on using Oracle Stream Analytics, Integrating with twitter, using database as reference.
Please install Oracle Stream Analytics (OSA) 12.2.1 from following URL. http://www.oracle.com/technetwork/middleware/complex-event-processing/downloads/index.html
OSA Installation is as simple as download and execute. You can refer to installation instruction available in OSA documentation. https://docs.oracle.com/middleware/12212/osa/using-streamanalytics/toc.htm
Once installtion is complete go to domain and start server by “startwlevs.cmd” or “startwlevs.sh” and Login to Oracle Stream Analytics (OSA) browser. http:// <hostname>:<port>/sx.
I have configured OSA on port 9202. Hence my URL is http://localhost:9202/sx. provide username and password.
To create connection with Twitter account
Go to Catalog tab. And select menu “Create New Item” -> “Connection”.
Provide connection name,description, tags. Select Connection type as Twitter.
Next screen it asks for Twitter connection dteail.
As you can see it asks Key & token to communicate with Twitter. For this Go to https://apps.twitter.com/
Click on Create New App. And provide information as shown below.
Agree terms & conditions and click on “Crate your Twitter application” button.
You will get message that application has been created successfully.
Now, access the application and go to “Keys and Access Tokens” section. It has key and token information unique to the application. Copy these values and paste in OSA connection page.
You will also need to generate Access Token. Click on the button “Create my access token”
Go back to OSA console and provide all the information it asked to integrate with Twitter.
Click save and your twitter connection is ready to use.
Let’s create a Stream that will read desired tweets.
Provide stream name, tags and source type as Twitter.
Provide hashtag you want to ready. Here I am using tweets from #sarkar3 and #sarkar3review
Click next and select default twitter shape. And click on create button.
Our stream is ready to use.
Next we will create an Exploration on the stream we created recently.
Goto Catalog. Click on Menu “Create New Item”-> Exploration.
Provide Exploration name, tags. Select Source as the stream we created above “Sarkar3 Movie Review”.
Click on Create. It will take to the exploration page.
As you can see it contains Stream in sources. If you scroll down it will start capturing live tweets in “Live output stream” section. As shown below.
Now to analyze sentiments we will need to filter tweets based on keywords. We will keep these keywords in database table.
Let’s create sentiment datastore. Goto your database and create a table as shown here.
I have created a table SENTIMENTS in my Oracle XE instance. With following columns and records.
And insert following records.
Next step is to create a datastore to access db table created above. For this login to Oracle Event processing (OEP) visualizer and create a datasource. Later we will use this datasource in stream analytics.
To open OEP visualize in your browser enter URL- localhost:9202/wlevs. You should see following screen.
Expand your OSA domain name and click on defaultserver.
Click on datasource tab. And click on Add button to create a new datasource.
Provide desired datasource name and JNDI name. Here I have used name as Demo.
Click on next tab “global transaction protocol”. Provide connection information. And save it.
You should see datasource created with successful deployment message.
Now come back to OSA console and click on Create new Item-> Connection.
Provide desired datasource name as “Sentiment data source”. And select source type as Database table.
Click Next. You should see following page. ‘demo’ datasource we created in OEP visualizer can be seen in the list. Select demo as data source name.
Click next. You should see list of tables. Select SENTIMENTS table from the list.
Click on save. Datasource is created successfully.
Now go back to Exploration and add “sentiment data source” in source section.
You will get error asking to correlate both source streams.
Click on Add a Correlation and select “lang” field from Twitter stream and “LANG” field from datasource table.
Add a filter to identify tweets matches to our key words defined in database.
You can remove columns from live output stream section. Just right click on the column and select remove.
Let’s tweet review on #sarkar3review or #sarkar3, hashtags we are monitoring.
Here are tweets with sentiments in OSA exploration.
We can further publish exploration results to a BAM dashboard or messaging queue for further analysis.
