ML Module Week 3 Topic 4: Logistic Regression

[EricThomson]

Logistic regression classification with scikit-learn.

Explain its role as a linear model that outputs a probability. Discuss interpretability (this is one strength of logistic regression). Evaluate using the metrics from Topic 2 (knn).

Setup for ANNs: Frame the lesson with the analogy that the concepts they are learning (weighted inputs, summation, activation function) are the same building blocks that make up a single neuron in a neural network. This will make the transition to the next lesson in Week 4 a bit more natural.

[EricThomson]

For general setup for the lesson -- iris dataset, digits dataset, and classifier evaluation metrics please see issue #59

Some resources I found useful on logistic regression (which is implemented nicely in scikit-learn):
https://www.youtube.com/watch?v=yIYKR4sgzI8
https://medium.com/%40roseserene/back-to-basics-logistic-regression-less-math-more-intuition-e473aebcf64a

I recommend start with a simple two-category case and then work on the cases for this lesson (iris/digits) as a natural code extension. Logistic regression is really most natural and clean for binary classification, so I think the thing to do is run it on iris dataset initially for two classes (e.g., setosa vs versicolor), show the probability surface, and decision boundary with just two classes to make it really simple., show how the as differente features change, the odds change.

Make point that logistic regression is drawing a LINEAR DECISION BOUNDARY (there is a way to draw decision boundaries in scikit-learn this will be really important for this case, you can use: sklearn.inspection.DecisionBoundaryDisplay). We will really want to drive home that logistic regression is only drawing lines between points (we can restrict to two input features to make this point even cleaner). This will lead to an important point in neural networks the following week, it isn't just a boring exercise : multilayer (deep) neural networks can solve problems that are not linearly separable. So this will be an important point moving forward. 😄

Once we make this point, go to N-class logistic regression. Honestly, it doesn't do that well (decision tree will do WAY better), but it is really good and interpretable for 2-category problems and has nice probability interpretation.