Description
Assignment #3
CSCI 5408 (Data Management, Warehousing, Analytics)
Problem #1: This problem contains three tasks.
Task 1: Cluster Setup – Apache Spark Framework on GCP
(if no GCP credit available – Hadoop or Spark setup in personal Linux machine)
Using your GCP cloud account, configure and initialize Apache Spark cluster.
Create a flowchart or write ½ page explanation on how you completed the task, include this part in
your PDF file.
Task 2: Data Extraction and Preprocessing Engine: Sources – NewsAPI
Steps for NewsAPI Operation
Step 1: Visit the news API https://newsapi.org/
Step 2: Create a developer account
Step 3: Search keywords – “Canada”, “University”, “Dalhousie”, “Halifax”, “Canada Education”,
“Moncton”, “Toronto”, “
Summer 2021 saurabh.dey@dal.ca
Step 3: Write a well-formed script/program using Java to extract data (Extraction Engine) from
NewsAPI.
(Do not use any online program codes or scripts, which is not part of the official API
documentation and specification.)
Step 4: You need to include an appropriate pseudocode of your data extraction program in the PDF
file.
Step 5: The captured raw data should be kept (programmatically) in files. Each file should not
contain more than 5 news articles. These files will be needed for “Problem #1-Task 3”
Step 6: Your program (Filtration Engine) should automatically clean and transform the data stored
in the files, and then upload each record to new MongodB database myMongoNews
o For cleaning and transformation -Remove special characters, URLs, emoticons etc.
o Write your own regular expression logic. You cannot use libraries such as, jsoup, JTidy
etc.
Step 7: You need to include a flowchart of Step 6 in the PDF file.
Task 3: Data Processing using Spark – MapReduce (written in Java) to perform count
Step 1: Write a MapReduce program (WordCounter Engine) to count (frequency count) the
following substrings or words. Your MapReduce should perform the frequency count on the stored
raw news files (titles and contents of the news articles)
o “Canada”, “Nova Scotia”, “education”, “higher”, “learning”, “city”, “accommodation”,
“price” – (case sensitive)
o You need to include a flowchart/algorithm of your MapReduce program on the PDF file.
Step 2: In your PDF file, report the words that have highest and lowest frequencies (it must
be computed programmatically).
Problem #2: This problem contains one task
Task 1: Data Visualization using Graph Database – Neo4j for graph generation
Step 1: Explore Neo4j graph database, understand the concept, and learn cypher query language
Step 2: Visit NovaScotia parks website that you used in Assignment 1.
Step 3: Using Cypher, create graph nodes with
• names of each region (e.g. Cape Breton Island Parks) as node, and
• names of parks as nodes.
You should add properties to the nodes. For adding properties, you should check the dataset
that you used in Assignment 1. E.g. location, street name, size etc. could be added as
properties
o All regions are parts of Nova Scotia, so all regions should be connected using edges.
o Each region has multiple parks, and therefore, there should be edges between parks and
the region.
o Once the graph is constructed on Neo4j – using cypher language, find which region has
more number of parks. Provide the screenshot on the PDF file.
o Include all your Cyphers (graph construction, find query etc.) and generated graph
image in the PDF file.
Summer 2021 saurabh.dey@dal.ca
Assignment 3 Submission Format:
1) Compress all your reports/files into a single .zip file and give it a meaningful name.
You are free to choose any meaningful file name, preferably – BannerId_Lastname_firstname_5408_A3
but avoid generic names like assignment-3.
2) Submit your reports only in PDF format.
Please avoid submitting .doc/.docx and submit only the PDF version. You can merge all the reports into
a single PDF. You should also include output (if any) and test cases (if any) in the PDF file.
3) Your executable code/script needs to be submitted on https://git.cs.dal.ca/
