CSE 587 Spring 2023: Deep Learning for Natural Language Processing

Course Information

Instructor Lecture Time and Location TA Course Syllabus

Course Goals and Objectives

Students will gain necessary skills and experience to understand, design, implement, and test their own NLP models using neural networks through programming assignments and a final project. After successfully completing this course, students will be able to:
  1. Design, implement, and test NLP models based neural networks
  2. Analyze and assess the performance of NLP models
  3. Situate their research contributions with reference to the state-of-the-art
  4. Present their results in an academic fashion including both research papers and oral presentations


Since this course centers on deep learning methodology for NLP, CMPSC 448 Machine Learning or CSE 582 Natural Language Processing is the prerequisite. This course also assumes programming skills in Python and knowledge in linear algebra, calculus, basic probability and statistics.


The following textbooks are recommended for reading beyond papers listed in the course schedule. All of them are publicly available!

Course Project

Project Format. This project aims to conduct original and independent research over NLP-related topics. Students can choose from several possible approaches:
  1. Invent a new and important task in NLP and create a dataset for it, e.g., MACSum: Controllable Summarization with Mixed Attributes.
  2. Create a new dataset for an existing NLP tasks, e.g., Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.
  3. Pick an existing NLP task and dataset and try to get good results, e.g., CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning and check this awesome website Papers With Code: The latest in Machine Learning.
  4. Try to pick a NLP problem and dive deep into it with comprehensive analysis, e.g., Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.
No matter which approach you take, we will hold your project to a high standard. Excellent projects will result in published papers in top-tier NLP/AI/ML conferences or journals (e.g., ACL, NAACL, EMNLP, AAAI, ICLR, NeurIPS, ICML, TACL, ...).

Group Policy. You can work on the course project in a group of 1-3 people. You are allowed to combine this project with your research projects or projects from other courses.

  1. Project Proposal. Write a 3-page proposal that outlines your plan including what problem or task you want to address, what dataset(s) you want to work on, what metrics you need to employ, what baselines you would like to compare with. You should also cite a few relevant prior papers. Please use this overleaf templatet.
  2. Final Report. Your final report should use our Latex template with at least 8-page plus references. Your report should begin with an abstract and introduction to clearly state the problem you want to solve and contributions you have made. It should also have a section on related work, a section on your methodology, a section on your experimental settings and results, and a section on conclusions. Please use this overleaf template.
  3. Code and Data. Please submit your data and runnable code with a detailed instruction.

Paper Presentation and Paper Review

In the second phase of this course, we will cover two paper presentations by students in one lecture.

Course Schedule

Date Topic Material Event Due
Part 1: Lectures by Rui on NLP Foundations
Week 1
Monday Jan 9
Wednesday Jan 11 Text Classification and Language Modeling
Week 2
Monday Jan 16
No Class
Wednesday Jan 18
Neural Networks and Backpropagation
Week 3
Monday Jan 23
Neural Networks and Backpropagation
Wednesday Jan 25 Recurrent Neural Networks, Sequence-to-Sequence, and Attention
Week 4
Monday Jan 30
Final Project
Assignment 1 Out
Wednesday Feb 1 Transformers
Week 5
Tue Feb 7
Project Group Registration Due
Wednesday Feb 8 BERT and Pretraining
Week 6
Monday Feb 13
BERT and Pretraining
Assignment 1 Due
Wednesday Feb 15 Prompt-based Methods
Week 7
Monday Feb 20
ChatGPT and Beyond
Project Proposal Due
Wednesday Feb 22 ChatGPT and Beyond
Part 2: Presentations by Students on NLP Frontiers
Week 8
Monday Feb 27
GPT-2 and GPT-3 Paper Presentations
Wednesday March 1 In-Context Learning Paper Presentations
Week 9
Monday March 6
Spring Break
Wednesday March 8 Spring Break
Week 10
Monday March 13
Calibration Paper Presentations
Wednesday March 15 Reasoning and Emergent Abilities Paper Presentations
Week 11
Monday March 20
Diffusion Model Paper Presentations
Wednesday March 22 Multilingual Language Model Paper Presentations
Week 12
Monday March 27
Multimodal Language Model Paper Presentations
Wednesday March 29 Code Language Model Paper Presentations
Week 13
Monday April 3
Language Models for RL Paper Presentations
Wednesday April 5 Knowledge Paper Presentations
Week 14
Monday April 10
Reinforcement Learning from Human Feedback Paper Presentations
Wednesday April 12 Task Generalization Paper Presentations
Week 15
Monday April 17
Evaluation Paper Presentations
Wednesday April 19 Data Paper Presentations
Week 16
Monday April 24
Security and Privacy Paper Presentations
Wednesday April 26 Social Impacts Paper Presentations
Friday May 5 Project Report, Code, and Data Due