Description
Objective
In this assignment, you will be designing and implementing streaming applications for performing basic
analytics on textual data. The data stream to be analyzed will be coming from the Twitter stream.
Streaming applications may seem complex but understanding how they operate is critical for a data
scientist. To allow you to explore a more complex implementation in a short period of time, you are
allowed to develop code based on already existing online code snippets and libraries with proper
attribution. In the scope of this assignment, you will learn:
• How to capture real-time data
• How to setup a stream processing pipeline
• How to process and get basic insights
• How to store the final processing results to a file
Important Notes:
• You must use the submit command to electronically submit your solution by the due date.
• All programs are to be written using Python 3.
• Your programs should be tested on the docker image that we provided before being submitted.
• To get full marks, your code must be well-documented.
What to Submit
When you have completed the assignment, move or copy your python scripts and outputs in a directory
(e.g., assignment3), and use the following command to electronically submit your files:
% submit 4415 a3
The team.txt file includes information about the team members (first name, last name, student ID,
login, yorku email). The readme.txt file includes step-by-step instructions on how to run each
application. You should also submit the python scripts (any *.py script) with representative names. In
addition, you should submit any output file (*.txt) that example runs of your application produce. You
can submit files individually after you complete each part of the assignment– simply execute the submit
command and give the filename that you wish to submit. You may check your submission‘s status by:
% submit -l 4415 a3
A. Identifying Trends in Twitter (60%)
Twitter is one of the main online social networks where users post and interact with messages known as
“tweets”. Tweets allow for instant, short, and frequent communication and they have been proved an
effective way to communicate news and other timely information. Therefore, a practical use for
Twitter’s functionality is to be used for identifying trends in real-time. Identifying trends is important for
several industries and services, including marketing, customer service, and crisis response.
Your task is to design and implement a Twitter streaming application that tracks specific hashtags and
reports their popularity (# occurrences) in real-time. In particular, you need to:
• Identify 5 related #hashtags (e.g., political parties, companies, product brands, stocks, etc.)
• Collect tweets mentioning any of the 5 #hashtags in real-time
• Compute the number of occurrences of each of the mentioned hashtags
• Plot the results of your analysis in real-time. Alternatively, you can decide to store the results in
a file, post-process them as a batch (offline) and create a plot based on the post-process
analysis. The results are based on the time window that your application is running (from the
time it begins, until it is killed or interrupted/stopped).
For the needs of your assignment, you will need to stitch together a number of technologies that can
enable the analysis to be performed, including:
A Twitter client: This is an application that connects to the Twitter service and obtains tweets as they
become available. It requires to create your own credentials to access the Twitter APIs. See Appendix A.
Apache Spark Streaming: This is an apache spark streaming application that connects to your twitter
client, receives the tweets as a stream, performs real-time processing of the incoming tweets, extracts
useful information, and computes the quantities of interest (i.e., number of occurrences of a #hashtag).
Real-time reporting: This is a visualization component that reports through a plot the results computed
by the apache spark streaming application in real-time. This can be implemented using AJAX
(asynchronous HTTP calls); see the resources of the Appendix B for examples. Alternatively, results can
be stored in a file in real-time, post-processed as a batch (offline), and presented as a plot.
Notes:
• Several implementation approaches exist for this application. You are encouraged to make
assumptions, make decisions and follow the technical path that you feel is more appropriate.
You will have a chance to explain your approach during the marking session
• Appendix A provides instructions on how to setup a Twitter application
• Appendix B provides several online resources related to the assignment
B. Real-time Sentiment Analysis of Twitter Topics (40%)
In the field of computational linguistics and natural language processing, sentiment analysis aims to
determine the attitude of a speaker, writer, or other subject with respect to some topic or the overall
contextual polarity or emotional reaction to a document, interaction, or event. A basic task in
sentiment analysis is classifying the polarity of a given text at the document, sentence, or
feature/aspect level—whether the expressed opinion in a document, a sentence or an entity
feature/aspect is positive, negative, or neutral. For example, consider the following three text inputs:
“I love ice cream a lot”
“I dislike ice cream a lot”
“ice cream is made from milk”
One would expect that the polarity of the first is (rather) positive, of the second is (rather) negative and
of the third is (rather) neutral. The word “rather” is used here to express subjectivity, since humans not
always agree about the polarity of a sentence. We rely on an out-of-the-shelf library to perform
sentiment analysis. The analysis will be on the level of a document, where the document is a tweet (i.e.,
all the words in a single tweet).
Your task is to design and implement a Twitter streaming application that performs sentiment analysis
of tweets related to competitive topics and provides a real-time monitoring of the polarity.
• Identify 5 competitive topics (e.g., political parties, companies, product brands, stocks, etc.)
• Manually select a set of 10 hashtags that better describe each of the topic identified above
• Collect tweets related to the 5 topics in real-time and perform sentiment analysis for each topic
• Plot the results of your analysis in real-time. Alternatively, you can decide to store results in a
file, post-process them and create a plot based on the post-process analysis
Notes:
• The implementation approach for the streaming application should be similar to the one
followed in Part A
• For the sentiment analysis you should employ Python’s Natural Language Toolkit (NLTK) library
• Appendix A provides instructions on how to setup a Twitter application
• Appendix B provides several online resources related to the assignment
Appendix A – Setting up a Twitter Application & Installing Tweepy
To start collecting tweets, you need to set up a Twitter application and get credentials that allow you to
pull tweets out of the twitter streaming API. Then, you need to develop a Twitter client that connects to
Twitter and acquires Twitter data. You can do that using Tweepy.
Create a Twitter Application and Obtain OAuth Access Keys
Briefly, you need to:
• Create a Twitter developer account: https://developer.twitter.com/en/apply-for-access
• Create a New Application
• Fill in your Application Details
o Name: Your app name. It needs to be a unique name across all twitter applications
o Description: A short description for your app
o Website: The website address where the app will be hosted. Use a placeholder for now
o Callback URL: Ignore this field
• Create Your Access Token
• Choose what Access Type You Need (choose ‘Read only’)
• Make a note of your OAuth Settings
Once you’ve done this, you will have the following OAuth settings.
• Consumer Key
• Consumer Secret
• OAuth Access Token
• OAuth Access Token Secret
You should keep these secret, since anyone with the keys, could effectively access your Twitter account.
Detailed information is provided here: https://docs.inboundnow.com/guide/create-twitter-application/
Install Tweepy
Tweepy is a python library for accessing the Twitter API. You can install Tweepy using pip:
$pip install tweepy
You may also use Git to clone the repository directly from Github and install it manually:
$git clone https://github.com/tweepy/tweepy.git
$cd tweepy
$python setup.py install
The next step is to use Tweepy to create a Twitter application that uses your Twitter credentials.
More information: https://github.com/tweepy/tweepy
Appendix B – Useful Online Resources and Tutorials
Spark Streaming Programming Guide
https://spark.apache.org/docs/latest/streaming-programming-guide.html
Python Streaming Examples
https://github.com/apache/spark/tree/master/examples/src/main/python/streaming
An easy-to-use Python library for accessing the Twitter API
https://www.tweepy.org/
Apache Spark General Tutorial
https://www.toptal.com/spark/introduction-to-apache-spark
Apache Spark Streaming Tutorial: Identifying Trending Twitter Hashtags
https://www.toptal.com/apache/apache-spark-streaming-twitter
Twitter Trends Analysis using Spark Streaming
https://www.awesomestats.in/spark-twitter-stream/
Apache Spark Streaming with Twitter (and Python)
https://www.linkedin.com/pulse/apache-spark-streaming-twitter-python-laurent-weichberger/
Twitter-Sentiment-Analysis-Using-Spark-Streaming-And-Kafka
https://github.com/sridharswamy/Twitter-Sentiment-Analysis-Using-Spark-Streaming-And-Kafka
Twitter Sentiment Analysis using Python
https://www.geeksforgeeks.org/twitter-sentiment-analysis-using-python/
Sentiment Analysis on Reddit News Headlines with Python’s Natural Language Toolkit (NLTK)
https://www.learndatasci.com/tutorials/sentiment-analysis-reddit-headlines-pythons-nltk/
