CS 274A

CS 274A: Probabilistic Learning: Theory and Algorithms

Time: Winter Quarter 2013, Mondays and Wednesdays, 2:00 to 3:20pm
Location: ICS 174
Course code: 35540
Professor: Padhraic Smyth

Email: smyth at ics.uci.edu
Note: for all class-related emails please put [CS274] at the start of the subject line of your email

Office Hours: Mondays, 3:30 to 5:00pm, DBH 4212

Reader: Wei Ping, email: weiping.thu at gmail.com

Office Hours: Mondays, 5:00 to 6:00 pm, DBH 4069

For questions related to the homeworks please email Wei first

Class Notes:

Note Set 2: Multivariate Probability Models (PDF)

Note Set 3: Models, Parameters, and Likelihood (PDF)

Note Set 4: The EM Algorithm for Gaussian Mixtures (PDF)

Homeworks:

General information about MATLAB
Homework 1: [PDF] [LaTeX]
Homework 2: [PDF] [LaTeX] [Matlab code]
Homework 3: [PDF] [LaTeX]
Homework 4: [PDF] [LaTeX]
Homework 5: [PDF] [LaTeX] [spam email data]
Homework 6: [PDF] [LaTeX] [data sets]
Additional files for compiling LaTeX versions of the homeworks

Textbook:
Machine Learning: A Probabilistic Perspective, K. Murphy, MIT Press, 2012.

Additional Background/Reference Texts on Probability and Statistics

Bayesian Reasoning and Machine Learning, by David Barber, (PDF version freely available online)

Chapters 8, 9, 10, 11, 12, 13, 20 are particularly relevant for the 274A class

Probability: The Analysis of Data, vol 1, by Guy Lebanon (html version freely available online)

All of Statistics: A Concise Course in Statistical Inference, by Larry Wasserman

Course Goals: Students will develop a comprehensive understanding of probabilistic approaches to learning from data. Probabilistic learning is a key component in many areas within modern computer science, including artificial intelligence, data mining, speech recognition, computer vision, bioinformatics, and so forth. The course will provide a tutorial introduction to the basic principles of probabilistic modeling and then demonstrate the application of these principles to the analysis, development, and practical use of machine learning algorithms. Topics covered will include probabilistic modeling, defining likelihoods, parameter estimation using likelihood and Bayesian techniques, probabilistic approaches to classification, clustering, and regression, and related topics such as model selection and bias/variance tradeoffs. Knowledge of basic concepts in probability, multivariate calculus, and linear algebra are strongly recommended for this course. Homeworks will make use of the MATLAB programming environment (students will be expected to learn to use and program in MATLAB as part of this course)

Syllabus (subject to adjustment during the quarter):

Week	Monday	Wednesday	Reading
week 1: Jan 7th	Introduction Review of Probability: random variables, conditional and joint probabilities, Bayes rule, law of total probability, chain rule and factorization. Different interpretations of probability: frequentist and Bayesian views.	Multivariate Probability Models Working with sets of random variables. The multivariate Gaussian model. Independence, conditional independence, and graphical models.	Note Sets 1 and 2 Chapters 1 and 2.1 through 2.6 in text
week 2: Jan 14th	Learning from Data Concepts of models and parameters. Definition of the likelihood function and the principle of maximum likelihood parameter estimation.	Maximum Likelihood Learning I How to use maximum likelihood methods to learn the parameters of Gaussian models, binomial, multivariate and other parametric models.	Note Set 3 Chapter 10 in text, sections 10.1 through 10.3
week 3: Jan 21st	No Class University Holiday (Martin Luther King Day)	Maximum Likelihood Learning II	Chapter 4 in text, section 4.1 Optional Reading: Tutorial paper on maximum likelihood estimation
week 4: Jan 28th	Bayesian Learning: I General principles of Bayesian estimation: prior densities, posterior densities, MAP, fully Bayesian approaches. Beta/binomial	Bayesian Learning: II Bayesian estimation (ctd): estimation of Gaussian parameters.	Chapter 3.3/3.4 on Beta/Binomial and Dirichlet/Multinomial Chapter 5.2 on posterior distributions (Optional) Chapter 4.6 on estimating parameters of multivariate Gaussians
week 5: Feb 4th	Bayesian Learning: III Predictive densities, model selection, model averaging	Midterm exam (in class)	Chapter 3.2 on predictive densities and Chapter 5.3 onBayesian model selection.
week 6: Feb 11th	Classification I Introduction, Bayes rule, classification boundaries, discriminant functions	Classification II Optimal decisions, Gaussian classifiers, Bayes error rate discriminant functions	Chapter 3 in text, section 3.5 Chapter 4 in text, section 4.2 Optional Reading: naive Bayes models in spam email filtering
week 7: Feb 18th	No Class University Holiday (President's Day)	Classification III Class-conditional modeling. Likelihood-based approaches and properties of objective functions. Logistic regression and neural network models.	Barber text: pages 376 to 382 on logistic regression Murphy text: pages 246 to 271 Notes on logistic regression from Charles Elkan Logistic regression for high-dimensional text data
week 8: Feb 25th	Mixture Models and EM I K-means clustering. Mixtures of Gaussians and the associated EM algorithm. Clustering applications. Mixtures of conditional indepedence models.	Mixture Models/EM II The EM algorithm. Applications to text data. Underlying theory of the EM algorithm.	Note Set 4 Chapter 11 in text, sections 11.1 through 11.4 Optional Reading: Fraley and Raftery paper on model-based clustering Jeff Bilmes tutorial notes on EM Frank Dellaert's tutorial notes on EM Liang and Klein: Online EM with applications to text Manning and Schutze: document clustering Blei and Lafferty: review paper on topic modeling
week 9: Mar 4th	Regression Modeling I: Linear models. Normal equations. Systematic and stochastic components.	Regression Modeling II: Parameter estimation methods for regression. Maximum likelihood and Bayesian interpretations.	Chapter 7 in text, sections 7.1 through 7.3 and section 7.6 Optional Reading: - Andrew Ngs' video lectures on regression - Pages 1 to 33 of the Bias/Variance dilemma paper - Tipping's review paper on Bayesian regression
week 10: Mar 11th	State-space Models: hidden Markov and linear Gaussian models	Monte Carlo Methods: Importance sampling. Gibbs sampling and Markov Chain Monte Carlo (MCMC). Sequential sampling	Chapter 17 in text, sections 17.2 through 17.5 Chapter 23 and Chapter 24.1 and 24.2
finals week

Grading Policy
Final grades will be based on a combination of homeworks, exams, and projects: 25% homeworks, 35% midterm, and 40% final. Note that a requirement for this course is that students learn and use MATLAB for class homeworks.

Academic Honesty
Academic honesty is taken seriously. For homework problems or programming assignments you are allowed to discuss the problems or assignments verbally with other class members, but under no circumstances can you look at or copy anyone else's written solutions or code relating to homework problems or programming assignments. All problem solutions and code submitted must be material you have personally written during this quarter, except for any library or utility functions which we supply. Failure to adhere to this policy can result in a student receiving a failing grade in the class. It is the responsibility of each student to be familiar with UCI's academic honesty policies.

UCI Catalog Course Description:
Probabilistic Learning: Theory and Algorithms: A unified probabilistic framework for learning algorithms. Classical pattern recognition algorithms, probabilistic mixture models, kernel methods, hidden Markov models, among others. Multivariate data analysis concepts for classification and clustering. Methodologies such as cross-validation and bootstrap. Prerequisites: basic calculus and linear algebra.

CS 274A: Probabilistic Learning: Theory and Algorithms

Logistic regression for high-dimensional text data

Grading PolicyFinal grades will be based on a combination of homeworks, exams, and projects: 25% homeworks, 35% midterm, and 40% final. Note that a requirement for this course is that students learn and use MATLAB for class homeworks.

Grading Policy
Final grades will be based on a combination of homeworks, exams, and projects: 25% homeworks, 35% midterm, and 40% final. Note that a requirement for this course is that students learn and use MATLAB for class homeworks.