CS 4301: Numerical Methods for Machine Learning and Data ScienceFall 2020
An introduction to computational methods in linear algebra and numerical optimization methods with the aim of preparing students for higher level electives in data science, artificial intelligence, and machine learning. Learning outcomes:
- Ability to apply gradient descent to find local optima of differentiable functions
- Practical and theoretical understanding convex functions and convex sets
- Ability to solve constrained optimization problems via the method of Lagrange multipliers
- Ability to understand the precision issues that arise when applying iterative numerical methods to solve optimization problems
- Ability to formulate basic results in linear algebra as convex optimization problems (regression, projections, eigenvalues)
- Understand the differences, both practical and theoretical, between different matrix decomposition techniques
Schedule & Lecture Slides
|1||Aug. 17 & 19||Introduction Gradients and Multivariable Calculus|
|2||Aug. 24 & 26||Convex Sets and Convex Functions Gradient Descent||Boyd 2.1-2.3 Boyd 9.1-9.3 (can skip example subsections)|
|3||Aug. 31 & Sept. 2||Subgradient Methods Convex Optimization||Boyd 9.1-9.3 Boyd 4.1-4.3|
|4||Sept. 9||Projected Gradient|
|5||Sept. 14 & 16||Duality and Lagrange Multipliers||Boyd 5|
|6||Sept. 21 & 23||Constraint Qualification and KKT||Boyd 5|
|7||Sept. 28 & 30||Second Order Methods||Boyd 3.3, 9.5, 10.1-10.2|
|8||Oct. 5 & 7||ML Applications|
|9||Oct. 12 & 14||Linear Algebra ReviewPositive Semidefinite Matrices||Boyd A.5|
|10||Oct. 19 & 21||Eigenvectors, Eigenvalues, and Semidefinite Programming||Boyd A.5|
|11||Oct. 26 & 28||Singular Value Decomposition|
|12||Nov. 2 & 4||Midterm Discussion & More SVD|
|13||Nov. 9 & 11||Matrix Factorizations||CUR Decompositions|
|14||Nov. 16 & 18||Submodular Functions Alternating Projections|
|15||Nov. 23 & 25||Proximal Gradient Review|
All problem sets will be available on eLearning and are to be turned in there. See the homework guidelines below for homework policies.
Textbooks & References
There is no required textbook, but the following books may serve as useful references for different parts of the course.
- Convex Optimization by Stephen Boyd and Lieven Vandenberghe (online) .
- Convex Optimization Theory by Dimitri P. Bertsekas
All exams will be take home, open book, open notes. Midterm: 10/14, Take Home Final: TBD, during exam period
I expect you to try solving each problem set on your own. However, if you get stuck on a problem, I encourage you to collaborate with other students in the class, subject to the following rules:
- You may discuss a problem with any student in this class, and work together on solving it. This can involve brainstorming and verbally discussing the problem, going together through possible solutions, but should not involve one student telling another a complete solution.
- Once you solve the homework, you must write up your solutions on your own, without looking at other people's write-ups or giving your write-up to others.
- In your solution for each problem, you must write down the names of any person with whom you discussed it. This will not affect your grade.
- Do not consult solution manuals or other people's solutions from similar courses - ask the course staff, we are here to help!
UT Dallas Course Policies and Procedures
*adpated from David Sontag