Description
- How to download HTML pages from a website?
- How to extract relevant content from an HTML page?
Furthermore, you will gain a deeper understanding of the data science lifecycle.
Requirements:
- Please use pandas.DataFrame rather than spark.DataFrame to manipulate data.
- Please use BeautifulSoup rather than lxml to parse an HTML page and extract data from the page.
- Please follow python code style (https://www.python.org/dev/peps/pep-0008/). If TA finds your code hard to read, you will lose points. This requirement will stay for the whole semester.
Preliminary
If this is your first time to write a web scraper, you need to learn some basic knowledge of this topic. I found that this is a good resource: Tutorial: Web Scraping and BeautifulSoup.
Please let me know if you find a better resource. I’ll share it with the other students.
Overview
Imagine you are a data scientist working at SFU. Your job is to extract insights from SFU data to answer questions.
In this assignment, you will do two tasks. Please recall the high-level data science lifecycle below. I suggest that when doing this assignment, please remind yourself of what data you collected and what questions you tried to answer.
Figure 1. Data Science Lifecycle
Task 1: SFU CS Faculty Members
Sometimes you don’t know what questions to ask. No worries. Start collecting data first.
In Task 1, your job is to write a web scraper to extract the faculty information from this page: https://www.sfu.ca/computing/people/faculty.html.
(a) Crawl Web Page
A web page is essentially a file stored in a remote machine (called web server). Please write code to download the HTML page and save it as a text file (“csfaculty.html”).
# write your code
(b) Extract Structured Data
Please write code to extract relevant content (name, rank, area, profile, homepage) from “csfaculty.html” and save them as a CSV file (like faculty_table.csv).
# write your code
(c) Interesting Finding
Note that you don’t need to do anything for Task 1(c). The purpose of this part is to give you some sense about how to leverage exploratory data analysis (EDA) to come up with interesting questions about the data. EDA is an important topic in data science; you will learn it soon from this course.
First, please install dataprep. Then, run the cell below. It shows a bar chart for every column. What intersting findings can you get from these visualizations?
from dataprep.eda import plot
import pandas as pd
df = pd.read_csv("faculty_table.csv")
plot(df)
Below are some examples:
Finding 1: Professor# (29) is more than 3x larger than Associate Professor# (9).
Questions: Why did it happen? Is it common in all CS schools in Canada? Will the gap go larger or smaller in five years? What actions can be taken to enlarge/shrink the gap?
Finding 2: Homepage has 20.3% missing values.
Questions: Why are there so many missing values? Is it because many faculty do not have their own homepages or do not add their homepages to the school page? What actions can be taken to avoid this to happen in the future?
Task 2: Age Follows Normal Distribution?
In this task, you start with a question and then figure out what data to collect.
The question that you are intersted in is Does SFU CS faculty age follow a normal distribution?
To estimate the age of a faculty member, you can collect the year in which s/he graduates from a university (gradyear) and then estimate age using the following equation:
For example, if one graduates from a university in 1990, then the age is estimated as 2020+23-1990 = 53.
(a) Crawl Web Page
You notice that faculty profile pages contain graduation information. For example, you can see that Dr. Jiannan Wang graduated from Harbin Institute of Technology in 2008 at https://www.sfu.ca/computing/people/faculty/jiannanwang.html.
Please write code to download the 64 profile pages and save each page as a text file.
# Write your code
(b) Extract Structured Data
Please write code to extract the earliest graduation year (e.g., 2008 for Dr. Jiannan Wang) from each profile page, and create a csv file like faculty_grad_year.csv.
# write your code here
(c) Interesting Finding
Similar to Task 1(c), you don’t need to do anything here. Just look at different visualizations w.r.t. age and give yourself an answer to the question: Does SFU CS faculty age follow a normal distribution?
from dataprep.eda import plot
import pandas as pd
df = pd.read_csv("faculty_grad_year.csv")
df["age"] = 2020+23-df["gradyear"]
plot(df, "age")
Submission
Complete the code in this notebook, and submit it to the CourSys activity Assignment 1.
Bonus: Contribute to dataprep
Total Bonus: 0.2 + 0.3 = 0.5
- If you create an issue (i.e., report a bug or request a feature) at link1 or link2, you will get 0.2 bonus points.
- Creating more issues will not give you more bonus points, but you are encouraged to do so for learning more.
- If you send a pull request (i.e., fix a bug or implement a feature or add data connector for a new website) at link1 or link2 and the pull request gets merged into the repo, you will get 0.3 bonus points.
- Sending more pull requests will not give you more bonus points, but you are encouraged to do so for learning more.
- These bonus points will be directly added to your final grade.
How to Submit
- Submit github link(s) to the CourSys activity
Bonus. - Due on
March 15, 2020
If you love dataprep, please support it by simply clicking the Star on Github.