CS144 Project 3: Blogging Server on NodeJS and MongoDB solution

Description

Overview The primary task of Project 3 is to implement the website for our markdown-based blogging service that (1) lets anyone read blogs written by our users through public URLs and (2) lets our registered users create and update their own blogs after password authentication. Through this process, we will learn how to develop a back-end service using NodeJS, Express and MongoDB. Note that your implementation of Project 3 will be used for your Project 4 as well. Since Project 4 is dependent on Project 3, it is important that you follow instructions on this spec exactly to avoid any potential issues later in Project 4. Development Environment All development for Projects 3 (and 4) will be done on a Docker container based on the “junghoo/cs144-mean” image: $ docker run -it -p3000:3000 -p4200:4200 -v {host_shared_dir}:/home/cs144/shared –name mean junghoo/cs144-mean Make sure to replace {host_shared_dir} with the name of the shared directory on your host. The above command creates a docker container named mean with appropriate port forwarding and directory sharing. Once created, you can start the container simply by issuing the following command in a terminal window: $ docker start -i mean This container has MongoDB (v3.6.8), NodeJS (v8.12.0), Express application generator (v4.16.0), and Angular CLI (v7.0.3) pre-installed. Make sure that they run fine through the following commands: 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 2/14 $ mongo –version $ node –version $ express –version $ ng –version Project Requirements Our back-end blogging service should be accessible at the following URLs: # URL method functionality 1 /blog/:username/:postid GET Return an HTML-formatted page that shows the blog post with postid written by username. 2 /blog/:username GET Return an HTML page that contains first 5 blog posts by username. 3 /login GET, POST Authenticate the user through username and password. 4 /api/:username GET This is the REST API used to retrieve all blog posts by username 5 /api/:username/:postid GET, POST, PUT, DELETE This is the REST API used to perform a CRUD operation on the user’s blog post Note: 1. The URL patterns 1-3 should be publicly accessible by anyone. No prior user authentication should be required to access these URLs. More detailed requirements for the URL patterns 2 and 3 will be given in Parts B and C, respectively. 2. The URL patterns 4-5 should be protected behind authentication. More detailed specifications on this API will be given later in Part D. 3. The implemented server should listen on port 3000 for HTTP requests. All blog posts and the users’ authentication credentials should be stored in the MongoDB server. The MongoDB server should have at least the following two collections, “Posts” and “Users”, in the database “BlogServer”. The two collections must have two initial documents shown below: 1. Collection: Posts { “postid”: 1, “username”: “cs144”, “created”: 1518669344517, “modified”: 1518669344517, “title”: “Title 1”, “body”: “Hello, world!” } 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 3/14 { “postid”: 2, “username”: “cs144”, “created”: 1518669658420, “modified”: 1518669658420, “title”: “Title 2”, “body”: “I am here.” } The first collection “Posts” stores all blog posts created and saved by our users. As users write more blog posts, more documents should be inserted into this collection. Note that “created” and “modified” fields of the two documents are all integers, whose values are milliseconds since the the Unix epoch (Jan 1, 1970 UTC). 2. Collection: Users { “username”: “cs144”, “password”: “$2a$10$2DGJ96C77f/WwIwClPwSNuQRqjoSnDFj9GDKjg6X/PePgFdXoE4W6” } { “username”: “user2”, “password”: “$2a$10$kTaFlLbfY1nnHnjb3ZUP3OhfsfzduLwl2k/gKLXvHew9uX.1blwne” } The second collection “Users” stores the users’ authentication credentials. They should be used for authenticating any user to our server through the URL pattern 3. Note that users’ passwords must NEVER be stored in plaintext. Instead, we have to store them only after we apply a cryptographic one-way hash function. This ensures that even if a hacker breaks into our system and gets a hold of our database, they won’t be able to obtain the users’ passwords easily since it is time-consuming to recover the plaintext passwords from the hash values. The downside of this approach is that when a user tries to login, we will have to apply the same cryptographic hash function to the user-provided password and then match the equivalence of this hash value to what is stored in our database. This can potentially increase the computational overhead of authenticating a user, but given the potential security risk of saving plaintext passwords, it is the cost that we are willing to pay. In our case, we applied bcrypt hash function to each user’s password (“password” for “cs144” and “blogserver” for “user2”, respectively) using the bcryptjs module of node.js. Part A: Create Initial MongoDB Data In Project 3, all blog posts must be managed by MongoDB. Unlike MySQL, MongoDB doesn’t have the concept of schema. All types of data are saved as documents in a collection. Since MongoDB document is essentially a JSON object, it is often a preferred back-end data storage engine for JavaScript-based development. When you start the Docker container, it starts MongoDB server in the background. So you can start the “MongoDB command-line shell” simply by: $ mongo 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 4/14 Once you are inside the shell, you can issue most MongoDB commands interactively. Go over class notes on MongoDB to review the basic MongoDB commands. If needed, review online tutorials on MongoDB, such as this one. Now write a script named db.sh that includes the sequence of mongodb shell commands that load the following documents into the two collections, “Posts” and “Users”, in the “BlogServer” database: 1. Collection: Posts { “postid”: 1, “username”: “cs144”, “created”: 1518669344517, “modified”: 1518669344517, “title”: “Title 1”, “body”: “Hello, world!” } { “postid”: 2, “username”: “cs144”, “created”: 1518669658420, “modified”: 1518669658420, “title”: “Title 2”, “body”: “I am here.” } 2. Collection: Users { “username”: “cs144”, “password”: “$2a$10$2DGJ96C77f/WwIwClPwSNuQRqjoSnDFj9GDKjg6X/PePgFdXoE4W6” } { “username”: “user2”, “password”: “$2a$10$kTaFlLbfY1nnHnjb3ZUP3OhfsfzduLwl2k/gKLXvHew9uX.1blwne” } We also provide two JSON files that contain the above documents, posts.json and users.json, in case they are helpful. Note that “created” and “modified” fields of the first two post documents are all integers, whose values represent the milliseconds since the the Unix epoch (Jan 1, 1970 UTC). You must store all date fields in this format. Also recall that the above password values are obtained by applying the bcrypt cryptographic one-way hash function. Note that your provided script db.sh will be executed through the following command $ mongo < db.sh before we grade your submission to initialize the MongoDB database for the server. Note:： Your db.sh script must strictly follow the instructions above. Please make sure that the database names, the collection names, and their contents are exactly the same as this instruction including their case. Since MongoDB is CASE SENSITIVE, your submission will fail our test script even for a minor case mismatch and lead to a very low grade. Notes on CR/LF issue: If your host OS is Windows, you need to pay attention to how each line ends in your script file. Windows uses a pair of CR (carriage return) and LF (line feed) characters to terminate lines, but Unix uses only a LF character. Therefore, problems may arise when you feed a 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 5/14 text file generated from a Windows program to a Unix tool such as mongo. If you encounter any wired error when you run your script, you may want to run the dos2unix command in the container on your script file to fix line end characters. Part B: Implement Public HTML Blog Web Pages As the second task of Project 3, we now implement the public URLs by which any user can view blog posts published on our website. In particular, we will implement a server that returns an HTML page to an HTTP request to the following URL patterns: # URL method functionality 1 /blog/:username/:postid GET Return an HTML-formatted page that shows the blog post with postid written by username. 2 /blog/:username GET Return an HTML page that contains first 5 blog posts by username. If the user has more than 5 posts, the page should contain a “next” button that link to the next 5 posts (according to the postid) by the user. We use node.js JavaScript runtime engine and its express module to implement our back-end server. Node.js is built on Chrome’s V8 JavaScript engine, which uses an event-driven, non-blocking I/O model to make it lightweight and efficient. Express is a module that provides an easy-to-use routing mechanism, HTML template integration, and third-party middleware integration. You may want to go over the class lecture note on node.js to brush up on node.js and express. If you need more detailed instruction on how to use them, online tutorials like this can be helpful. Express web site has more detailed documentation on routing and using a template engine. Generate Server Skeleton Code To generate the skeleton code for our web server, we will use the Express application generator. Initialize the project under a project directory (e.g. blog-server): $ express -e blog-server Here -e option makes the generated code use the “EJS” template engine for generating HTML pages from JSON. You are welcome to use other template engine for your development, but our project spec gives instructions for “EJS”. When the above command is executed, you see the following folder and file structure within the blog-server directory: 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 6/14 blog-server +- bin +- public +- routes +- views +- package.json +- app.js Here the app.js file serves as the entrance of the whole project. If you define other .js files, you need to reach them starting from app.js. The package.json file contains some meta information on the project including its “package dependencies”. When you add new dependencies to this file and run npm install, npm will install all dependent modules into the subdirectory node_modules. The public directory contains static resources that are made available on the developed web site, like HTML, CSS, JavaScript, and image files. The views directory contains HTML template files. The routes directory contains “middleware” that processes the requests. The bin directory contains the executable file. As you develop your code, you are welcome to add or modify any directory or file as needed. Make sure that the generated code works properly on our container by executing the following command: $ npm start The npm start command executes “test” script specified in package.json (which is node ./bin/www in the auto-generated package.json file). Open a web browser on your host machine and make sure that you see a page similar to the following image at https://localhost:3000/: Install mongodb and commonmark Modules 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 7/14 To generate public HTML blog pages using node.js, express, and MongoDB, we need a MongoDB client library for node.js and a markdown-to-HTML rendering library. We will use the official MongoDB driver for node.js and the commonmark.js library for this purpose. Install these two packages using npm $ npm install –save mongodb $ npm install –save commonmark Make sure that you add the –save option, so that the packages are added as dependencies to package.json. Note that commonmark.js’s API is almost identical to the Java version, so you will find it easy to use. In case you are not familiar with the MongoDB native client API, go over MongoDB native drive quick start tutorial and read the MongoDB tutorial on CRUD operations to learn the basics. Note: Due to a strange interaction between node, Docker container, and shared folder, you may sometimes get an error similar to the following when you run node or npm: path.js:1177 cwd = process.cwd(); ^ Error: ENOENT: no such file or directory, uv_cwd If you get the above error, you can “fix it” by cd ../, cd back, and try again. Learn the Template Syntax The responses to the two URL patterns shown above should be all in HTML. For generating an HTML response, you can use a template engine. When you use a template engine, you just need to write a static “template file”, and “render” the final HTML response by combining the template with data. EJS (Embedded JavaScript) uses a template syntax very similar to Java ServePages (JSP). Like JSP, you can use the standard HTML tags in the template, and sprinkle your JavaScript code inside <% … %>, <%= … %>, or <%- … %> tags. <%= … %> or <%- … %> can include any expression and is replaced with the output string of the expression. The difference between the two is that <%= … %> escapes HTML tags in the output, so that the HTML tags are displayed as strings, while <%- … %> does not escape HTML tags, so that the browser can interpret them as HTML tags. <% … %> can include any arbitrary JavaScript code, not just expressions. Here is an example of a valid EJS template: 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 8/14 For more EJS examples, look at the generated template files in the views directory. You may also want to go over an online tutorial like this to learn more on EJS. Implement /blog/:username/:postid and /blog/:username Now that you know the basics to implement Part B, go ahead and implement it. The responses from these two URL patterns must meet the following requirements: 1. All blog posts returned from the two URL patterns should be rendered in HTML from markdown using the commonmark.js module, both title and body. 2. The second URL pattern /blog/:username must return the first 5 posts by username. When there are more posts from the user the returned page must contain a “next” link, which points to a page with the next 5 posts according to the postid by the user. Make sure that the “next” link is implemented as an HTML  element with id=”next” and href pointing to the URL of the “next page”. 3. The second URL pattern /blog/:username must take an optional query string parameter start=:postid, like /blog/cs144?start=3 When this optional query string parameter exists, the response must include the next 5 posts by cs144 whose postid is 3 or above. Note: 1. In implementing this part, remember that a request can be “routed” to a callback function through app.METHOD(URL, callback), like app.get(‘/blog/:username’, callback). Inside the callback function, you can reference the HTTP request through the first req parameter, and you can generate the response through the second res parameter. 2. In our project, all dates are stored as a number in MongoDB, which represents milliseconds since the the Unix epoch (Jan 1, 1970 UTC). You can convert a JavaScript Date object to this number using its getTime() method. Conversely, you can convert this number to a Date object by passing it as the constructor parameter of Date or by calling a date object’s setTime() method.

• <% for(let i = 0; i < books.length; i++) { %>
  • <%- books[i].isbn %>

<% } %>

 3. Please remember that all MongoDB commands must be executed asynchronously either using callback functions or using a Promise object (with await keyword). 4. To minimize the overhead from creating a connection to the database server, we strongly recommend that you create a connection to the MongoDB server when the your application starts up and reuse the created connection for all MongoDB commands. If you are unclear about how this can be done, see this page (This page is a slightly modified version of the original blog post published here.) Note that if you decide to add app.listen() to app.js file, you will have to remove the call to listen() from the “bin/www” file. Part C: Implement User Login Page In this part, we will implement the user login page available at the following URL: # URL method functionality 1 /login GET, POST Authenticate the user through username and password. Here are more detailed descriptions of the above API: 1. GET /login: The request may optionally include redirect=:redirect parameter. Given the request, the server should return an HTML page that contains an HTML form with at least two input fields, username and password. When the user inputs their username and password in these fields, and presses the submit button, the page should issue a request POST /login with username=:username&password=:password&redirect=:redirect, where the redirect parameter is added only if it was included as part of this GET request. 2. POST /login: The request body must contain username=:username&password=:password parameters, and optionally the redirect=:redirect parameter. When the provided username and password match our record, the server must (1) set an authentication session cookie in JSON Web Token (JWT) (more on this later) and (2a) redirect to redirect if it was provided in the request or (2b) return status code “200 (OK)” with the body saying that the authentication was successful. If the records do not match, the server must return the status code “401 (Unauthorized)” and an HTML form with username and password input fields in the response body. Recall that our MongoDB server stores all users’ credentials in the “Users” collection of the “BlogServer” database. In addition, recall that the stored user passwords are all hash values obtained by applying the bcrypt hash function. Therefore, to match a user’s stored password against what she provides during authentication, you will need to use node.js bcryptjs module. Install the module through the following command: $ npm install –save bcryptjs 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 10/14 Read the bcrypt package page to learn how you can use it to compare a user’s password against a hash value. JSON Web Token (JWT) Once the user’s authenticity is established through the user-provided password, our server must establish a “authenticated session”, so that it can recognize that any future request coming from the same browser comes from the authenticated user. There are a number of ways to implement this. In this project, you must use JSON Web Token (JWT) for this purpose. If you are not familiar with JWT, go over the JWT introduction page to learn what it is and how it can be used. In your implementation, once the user is authenticated, you must set a transient session cookie (a cookie that has no expiration date, so that it is forgotten once the browser is closed) whose name is jwt and whose value is the following JWT: 1. Its header must have two claims, ‘alg’ (algorithm) and ‘typ’ (type), with the following values: 2. Its payload must have two claims, ‘exp’ (expiration time) and ‘usr’ (user) where expiration should be two hours from now (in seconds since Unix epoch, Jan 1, 1970) and username should be the authenticated username. Note that the unit of time here is seconds not milliseconds. According to JWT standard, this is how the expiration time should be represented. 3. The signature must be generated using the HS256 algorithm (HMAC-SHA256 hash function) with the following secret key C-UFRaksvPKhx1txJYFcut3QGxsafPmwCY6SCly3G6c Once this JWT cookie is set, our server will be able to recognize that any future request from the browser comes from the authenticated user username. You can use the jsonwebtoken module to construct a JWT. Install it with the following command { “alg”: “HS256”, “typ”: “JWT” } { “exp”: expiration, “usr”: “username” } 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 11/14 $ npm install –save jsonwebtoken and learn how to use it by going over examples in the jsonwebtoken module page. Now implement the login page. Remember that after a successful authentication by the user (i.e., the user’s username and password match), the server should generate an appropriate JWT, set it as the value of the cookie jwt, and redirect the request to the redirect query parameter in the request. If the authentication fails, (username or password is missing or they don’t match with our record), the server must return a page with the username and password input box, so that they user can try again. Remember that the user “cs144”’s password is “password” and “user2”’s password is “blogserver” in testing your authentication page. Part D: Implement Blog-Management REST API In this part, you will have to implement the REST API that will be used by an Angular-based editor (that will be implemented in Project 4) to save, retrieve, update, and delete the blog posts from the server. # URL method functionality 1 /api/:username GET This is the REST API used by the Angular blog editor to retrieve all blog posts by username 2 /api/:username/:postid GET, POST, PUT, DELETE This is the REST API used by the Angular blog editor to perform a CRUD operation on the user’s blog post Here are more detailed descriptions of the above API: 1. GET /api/:username: The server should return all blog posts by username. The returned posts should be included in the body of the response as an array in JSON even if the user has zero or one post. Each post in the array must have at least five fields, postid, title, body, created, and modified (case sensitive). The response status code should be “200 (OK)”. 2. GET /api/:username/:postid: The server should return the blog post with postid by username. If such a post exists, the response status code should be “200 (OK)”, and the post should be included in the body of the response in JSON with at least four fields, title, body, created, and modified (case sensitive). If not, the response status code should be “404 (Not found)”. 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 12/14 3. POST /api/:username/:postid: When the server gets this request, it must insert a new blog post with username, postid, title, and body from the request. The request must include title and body in its body in JSON. The created and modified fields of the inserted post should be set to the current time. If the insertion is successful, the server should reply with “201 (Created)” status code. If a blog post with the same postid by username already exists in the server, the server should not insert a new post and reply with “400 (Bad request)” status code. 4. PUT /api/:username/:postid: The request must include title and body in its body in JSON. When the server gets this request, it must update the existing blog post with postid by username with the title and body values from the request. The modified field should be updated to the current time as well. If the update is successful, the server should reply with “200 (OK)” status code. If there is no blog post with postid by username, the server should reply with “400 (Bad request)” status code. 5. DELETE /api/:username/:postid: When the server gets this request, the server must delete the existing blog post with postid by username from the database. If the deletion is successful, the server should reply with “204 (No content)” status code. If there is no such post, the server should reply with “400 (Bad request)” status code. 6. All dates transmitted should be in milliseconds since the the Unix epoch (Jan 1, 1970 UTC) in number type. 7. If a request does not meet our requirements (such as not formatting data in JSON, not including required data, etc.), the server must reply with “400 (Bad request)” status code. 9. This REST API must be protected behind authentication. That is, if the request to this API does not contain a valid jwt cookie with matching username (i.e., if the jwt cookie is not included in the HTTP header, if the included jwt has expired, or if the username in jwt does not match the username in the URL), the server must reply with “401 (Unauthorized)” status code. Now add the appropriate routing instruction to your app and implement the above API. Make sure that the implementation works as intended before you proceed. You can test your implementation using the curl command (available within our Docker container) or Postman, which makes it easy to send an HTTP request to a server. Once you made sure that everything works fine, protect this REST API behind authentication. That is, before you process any request to this API, first make sure that the request contains a valid jwt cookie with the matching username. Note: Please make sure that you return the correct status code according to our spec. If your status code is diffrent from our spec you will get zero point for the part, even if the response body may contain reasonable content. What to Submit 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 13/14 Before you create the submission zip file for Project 3, please make sure that your server can be executed simply by running the following sequence of commands at the project root directory: $ mongo < db.sh $ npm start Since this is how our grader will run and test your code, it is essential that your code runs fine without any problem through the above commands. You may assume that the MongoDB server is running with no documents inside the “BlogServer” database when your your server is executed in the grader’s machine. After you have checked there is no issue with your project, you can package your work by running our packaging script. Please first create the TEAM.txt file. This file must include the 9-digit university ID (UID) of every team member, one UID per line. No spaces or dashes. Just 9-digit UID per line. If you are working on your own, include just your UID. Please make sure TEAM.txt, db.sh and package.sh are placed in the project­root directory, blog­server, like this: blog-server +- bin +- public +- routes +- views +- … +- package.json +- app.js +- db.sh +- package.sh +- TEAM.txt When you execute the packaging script like ./package.sh, it will check whether a few mandatory files are there, and package everything within the project directory (except the files in node_modules/) and create a file named project3.zip. If everything goes smoothly, you will see something like the following: [SUCCESS] Created ‘/home/cs144/shared/blog-server/project3.zip’, please submit it to CCLE. Please only submit this script­created project3.zip to CCLE. Do not use any other ways to package or submit your work! Grading Criteria 12/13/2018 index https://oak.cs.ucla.edu/classes/cs144/project3/index.html 14/14 The grading of Project 3 will be based on the following aspects: The server can work: You will get 0 if your server fails to work with npm start! Interaction with Database: The grader should be able to load the initial documents into the two collections using your db.sh script as is described in this spec. All the posts should be stored into and retrieved from MongoDB. Functionality: It should implement all the specified functions, i.e. list, insert, read, update and delete in the required way. You must implement REST APIs. URL Navigation: Users should be able to visit the corresponding contents by directly typing the corresponding URL as is described in this spec. Access Control: The HTMP pages must be public; while the REST API can be accessed only after login. Error Handling: Your server should handle different kinds of errors properly. Other Requirements in Description: For example, the next button in the /blog/:username page, which shows 5 posts each time.