| |
COL 870: Special Topics in Machine Learning - Deep Learning
(Semester II, 2020-21)
General Information
Semester: Sem II, 2020-21.
Instructor: Parag Singla (email: parags AT cse.iitd.ac.in)
Class Timings/Venue:
- Slot C. Tuesday, Wednesday, Friday, Saturday (selected): 8 am - 9 am.
- Venue: Online.
Click here
Teaching Assistants:
Yatin Nandwani, Arnab Mondal, Ashutosh Agarwal
Registration Details (including Assignment Teams):
Click Here
Sign up for Piazza
Code: As announced over email.
Pre-requisites
A foundational course in AI or ML.
Announcements
[Apr 15, 2021]: Assignment 2 is out. Due: Thursday May 6, 2021 (11:50 pm). To be done in pairs of 2.
[Mar 26, 2021]: Assignment 1 is out. Due: Wednesday April 15, 2021 (11:50 pm). To be done in pairs of 2.
[Feb 8, 2021]: Class venue link has been updated. Please use the updated link (posted above).
[Feb 8, 2021]: Class notes/videos for the first week are up on the website.
Obective:
This course is meant to be the first graduate level course in deep learning. Deep Learning is
an emerging area of Machine Learning which has revolutionized the progress in the field during
last few years with applications found in NLP, Vision and Speech to name a few domains. This
course is intended to give a basic overview of the mathematical foundations of the field, and
present the standard techniques/arhitectures which become basis for more advanced ones. About a
3rd of the course will focus on latest research topics in the area. Without an implementation, no
deep learning class can be complete. Students will get to implement some of the architectures on
a GPU to test on large datasets.
Content: Basics: Introduction. Multi-layered Perceptrons. Backpropagation. Regularization:
L1-L2 Norms. Dropouts. Optimization: Challenges. Stochastic Gradient Descent. Advanced Optimization
Algorithms. Convolutional Networks (CNNs). Recurrent Architectures. Dropout, Batch Normalization.
Generative Architectures. Advanced Architectures for Vision. Advanced Architectures for NLP.
More Recent Advances in the field.
Week-Wise Schedule
NOTE: The exact list of topics below is tentative (until we are past that week).
We will update it as we go through the lectures in each week. So, stay tuned!
| Week | Topic | Book Chapters | Supplementary Material |
| 1 | Introduction, Motivation | |
|
| 2 | Mulit-layered Perceptrons | Goodfellow et al. Chapter 6 |
|
| 3 | Backpropagation | Goodfellow et al. Chapter 6 |
|
| 3 | Regularization - L1/L2 | Goodfellow et al. Chapter 7 |
|
| 4 | Regularization - Other Techniques | Goodfellow et al. Chapter 7 |
|
| 5 | Optimization - Basics | Goodfellow et al. Chapter 8 |
|
| 6 | Optimization - Advanced Algorithms | Goodfellow et al. Chapter 8 |
|
| 7 | Deep Learning for Vision | Goodfellow et al. Chapter 9 |
|
| 8 | Deep Learning for Vision (Architectures) | |
See list of papers |
| 9 | Deep Learning for NLP | Goodfellow et al. Chapter 10 |
|
| 10 | Deep Learning for NLP (Attention) | |
See list of papers |
| 11 | Generative Models | |
See class notes, list of papers |
| 12 | Graph Neural Networks | |
|
| 13 | Deep Reinforcement Learning | |
|
| 14 | Advanced Topics | |
|
Notes:
Feb 3,Feb 5,
Feb 6,Feb 9,
Feb 10, Feb 12,
Feb 13,Feb 16,
Feb 17,Feb 23,
Feb 24,Feb 26,
Feb 27, Mar 2,
Mar 3, Mar 5,
Mar 6,Mar 9,
Mar 10,Mar 12,
Mar 23, Mar 24,
Mar 31, Apr 3,
Apr 6, Apr 7,
Apr 9, Apr 10 (Guest Lecture),
Apr 13 (Guest Lecture),Apr 14 (Guest Lecture),
Apr 16 (Guest Lecture),Apr 30,
May 4, May 5
Videos:
Feb 3,
Feb 5,
Feb 6,
Feb 9,
Feb 10,
Feb 12,
Feb 13,
Feb 16,
Feb 17,
Feb 23,
Feb 24,
Feb 26,
Feb 27,
Mar 2,
Mar 3 ,
Mar 5,
Mar 6,
Mar 9,
Mar 10,
Mar 12,
Mar 23,
Mar 24,
Mar 31,
Apr 3,
Apr 6,
Apr 7,
Apr 9,
Apr 10 (Guest Lecture),
Apr 13 (Guest Lecture),
Apr 14 (Guest Lecture),
Apr 16 (Guest Lecture),
Apr 30,
May 4,
May 5
| Chapters 1 - 5, Goodfellow et al. |
- Deep Learning. Ian Goodfellow, Yoshua Bengio and Aaron Courville. 2016.
Assignment Submission Instructions
- You are free to discuss the problems with other students in the class. But the final solution/code that you
produce should come through your individual efforts.
- Required code should be submitted using Moodle Page.
- Honor Code: Any cases of copying will be awarded a zero on the assignment. Additional
penalities will be imposed based on the severity of copying. Any copying cases run the chances of being
escalated to the Department/DISCO.
- Late policy: You are allowed a total of 5 buffer days for the three
programming assignments. There is no penalty if your submission stays withing the limit of the 5 buffer days
(total). For each additional day beyond the allowed 5 buffer days, you will lose 10% of the score for every
late day in submission.
Assignments
- Assignment 2. Due Date: Thursday May 06, 2021. 11:50 pm.
- Assignment 1 [updated with details of both Parts 1 and 2]. Due Date:
WednesdayThursday April 15, 2021. 11:50 pm.
- Assignment 0. Due Date: Wednesday Mar 10, 2021. 11:50 pm.
| Assignment 0 | 3% |
| Assignment 1 | 15% |
| Assignment 2 | 18% |
| Minor | 25% |
| Major (Tentative) | 40% |
|