CSE 587 Spring 2023: Deep Learning for Natural Language Processing

Course Information

Instructor

Rui Zhang
Office Hour: Wednesdays 4-6pm @ W329 Westgate Building and Zoom

Lecture Time and Location

Mondays and Wednesdays 2:30pm - 3:45pm @ Walker Building 124

Rishabh Bhatt
Office Hour: Thursdays 12-2pm @ W300

Course Syllabus

Detailed syllabus is avaialbe here.

Course Goals and Objectives

Students will gain necessary skills and experience to understand, design, implement, and test their own NLP models using neural networks through programming assignments and a final project. After successfully completing this course, students will be able to:

Design, implement, and test NLP models based neural networks
Analyze and assess the performance of NLP models
Situate their research contributions with reference to the state-of-the-art
Present their results in an academic fashion including both research papers and oral presentations

Prerequisites

Since this course centers on deep learning methodology for NLP, CMPSC 448 Machine Learning or CSE 582 Natural Language Processing is the prerequisite. This course also assumes programming skills in Python and knowledge in linear algebra, calculus, basic probability and statistics.

Textbook

The following textbooks are recommended for reading beyond papers listed in the course schedule. All of them are publicly available!

Speech and Language Processing by Dan Jurafsky and James H. Martin.
Dive into Deep Learning by Aston Zhang, Zack C. Lipton, Mu Li, Alex J. Smola.
Natural Language Processing by Jacob Eisenstein.
Foundations of Statistical Natural Language Processing by Chris Manning and Hinrich Schütze.
Linguistic Structure Prediction by Noah A. Smith.
Machine Learning: A Probabilistic Perspective by Kevin Murphy.
Linguistic Fundamentals for NLP by Emily M. Bender.

Course Project

Project Format. This project aims to conduct original and independent research over NLP-related topics. Students can choose from several possible approaches:

Invent a new and important task in NLP and create a dataset for it, e.g., MACSum: Controllable Summarization with Mixed Attributes.
Create a new dataset for an existing NLP tasks, e.g., Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.
Pick an existing NLP task and dataset and try to get good results, e.g., CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning and check this awesome website Papers With Code: The latest in Machine Learning.
Try to pick a NLP problem and dive deep into it with comprehensive analysis, e.g., Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.

No matter which approach you take, we will hold your project to a high standard. Excellent projects will result in published papers in top-tier NLP/AI/ML conferences or journals (e.g., ACL, NAACL, EMNLP, AAAI, ICLR, NeurIPS, ICML, TACL, ...).

Group Policy. You can work on the course project in a group of 1-3 people. You are allowed to combine this project with your research projects or projects from other courses.

Deliverables.

Project Proposal. Write a 3-page proposal that outlines your plan including what problem or task you want to address, what dataset(s) you want to work on, what metrics you need to employ, what baselines you would like to compare with. You should also cite a few relevant prior papers. Please use this overleaf templatet.
Final Report. Your final report should use our Latex template with at least 8-page plus references. Your report should begin with an abstract and introduction to clearly state the problem you want to solve and contributions you have made. It should also have a section on related work, a section on your methodology, a section on your experimental settings and results, and a section on conclusions. Please use this overleaf template.
Code and Data. Please submit your data and runnable code with a detailed instruction.

Paper Presentation and Paper Review

In the second phase of this course, we will cover two paper presentations by students in one lecture.

Please sign up for 1 paper presentation and 5 paper reviews by putting your names in [slot]. First come, first served. You have to review papers which are not presented on your presentation day.
Please submit your presentation slides and paper reviews in Canvas -> Assignment by the end of day before the class, i.e., 11:59pm Sunday for Monday's classes and 11:59pm Tuesday for Wednesday's classes.
Before your presentation, you are also required to meet me before the class: 1pm-2pm for Monday and 1pm-2pm for Wednesday.
Each presentation will be 35mins (e.g., 25mins + 10mins Q&A).
Please follow the review form from ACL for paper review Becoming a good reviewer takes efforts and practice. If you haven't review an NLP paper before, here are guidelines for review a paper provided by community.
- How to review for ACL Rolling Review
- Review Form

Course Schedule

Date	Topic	Material	Event	Due
Part 1: Lectures by Rui on NLP Foundations
Week 1 Monday Jan 9	Introduction [slides]
Wednesday Jan 11	Text Classification and Language Modeling [slides]
Week 2 Monday Jan 16	No Class
Wednesday Jan 18	Neural Networks and Backpropagation [slides]
Week 3 Monday Jan 23	Neural Networks and Backpropagation [slides]
Wednesday Jan 25	Recurrent Neural Networks, Sequence-to-Sequence, and Attention [slides]
Week 4 Monday Jan 30	Final Project [slides]		Assignment 1 Out
Wednesday Feb 1	Transformers [slides]
Week 5 Tue Feb 7	Transformers [slides]			Project Group Registration Due
Wednesday Feb 8	BERT and Pretraining [slides]
Week 6 Monday Feb 13	BERT and Pretraining [slides]			Assignment 1 Due
Wednesday Feb 15	Prompt-based Methods [slides]
Week 7 Monday Feb 20	ChatGPT and Beyond [slides]			Project Proposal Due
Wednesday Feb 22	ChatGPT and Beyond [slides]
Part 2: Presentations by Students on NLP Frontiers
Week 8 Monday Feb 27	GPT-2 and GPT-3	Paper Presentations LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Presented by Songhe Wang [slides] Language Models are Unsupervised Multitask Learners Presented by Mahsa Sheikhi [slides] Language Models are Few-Shot Learners Presented by Shu Zhao [slides]
Wednesday March 1	In-Context Learning	Paper Presentations Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? Presented by Renze Lou [slides] Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity Presented by Sarkar Snigdha Sarathi Das [slides]
Week 9 Monday March 6	Spring Break
Wednesday March 8	Spring Break
Week 10 Monday March 13	Calibration	Paper Presentations Language Models (Mostly) Know What They Know Presented by Yusen Zhang [slides]
Wednesday March 15	Reasoning and Emergent Abilities	Paper Presentations Chain of Thought Prompting Elicits Reasoning in Large Language Models Presented by Haoran Zhang [slides] Emergent Abilities of Large Language Models Presented by Malyaban Bal [slides]
Week 11 Monday March 20	Diffusion Model	Paper Presentations Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding Presented by Xinjie Li [slides] Diffusion-LM Improves Controllable Text Generation Presented by Max Mehta [slides]
Wednesday March 22	Multilingual Language Model	Paper Presentations Cross-lingual Language Model Pretraining Presented by Samira Malek [slides] Few-shot Learning with Multilingual Language Models Presented by Abdullah Al Ishtiaq [slides]
Week 12 Monday March 27	Multimodal Language Model	Paper Presentations Learning Transferable Visual Models From Natural Language Supervision Presented by Liwei Che [slides] Flamingo: a Visual Language Model for Few-Shot Learning Presented by Pouria Mahdavinia [slides]
Wednesday March 29	Code Language Model	Paper Presentations Evaluating Large Language Models Trained on Code Presented by Srinivas V [slides] InCoder: A Generative Model for Code Infilling and Synthesis Presented by Levent Toksoz [slides]
Week 13 Monday April 3	Language Models for RL	Paper Presentations ReAct: Synergizing Reasoning and Acting in Language Models Presented by Berk Atil [slides]
Wednesday April 5	Knowledge	Paper Presentations Language Models as Knowledge Bases? Presented by Huaisheng Zhu [slides] Fast Model Editing at Scale Presented by Allen Gao [slides]
Week 14 Monday April 10	Reinforcement Learning from Human Feedback	Paper Presentations Training language models to follow instructions with human feedback Presented by Ruihao Pan [slides] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Presented by Keaton Kraiger [slides]
Wednesday April 12	Task Generalization	Paper Presentations Multitask Prompted Training Enables Zero-Shot Task Generalization Presented by Alireza Sepehrinezhad [slides]
Week 15 Monday April 17	Evaluation	Paper Presentations Holistic Evaluation of Language Models Presented by Mohammad Sabih [slides]
Wednesday April 19	Data	Paper Presentations Large Language Models Can Be Strong Differentially Private Learners Presented by Zhimeng Guo [slides]
Week 16 Monday April 24	Security and Privacy	Paper Presentations Extracting Training Data from Large Language Models Presented by Hangfan Zhang [slides] Quantifying Memorization Across Neural Language Models Presented by Vishnu Dasu [slides]
Wednesday April 26	Social Impacts	Paper Presentations DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature Presented by Hamed Mahdavi [slides] A Watermark for Large Language Models Presented by Arisha Rao [slides]
Friday May 5				Project Report, Code, and Data Due