Classical ML
Logistic Regression
Connect a linear score to a sigmoid probability, a cross-entropy loss, and a decision threshold.
Jump to a section
Why this matters
Logistic regression turns a linear score into a probability with the sigmoid function. It is a classification model despite its name, and its parameters are commonly fitted by minimizing binary cross-entropy.
Distinguish the score, estimated probability, and decision threshold. Changing a threshold changes predictions without refitting the model. A linear decision boundary can still be useful, but it cannot express every relationship in the original features.
Build the mental model
The linear part draws a hyperplane through feature space. Points on one side score positive, the other negative. The sigmoid turns that raw score into a probability between 0 and 1, steeply near the boundary and saturating far from it. Predict class 1 when (i.e. when the linear score is positive).
A score becomes a probability
Read the probability at zero, then compare equally sized changes near the center and in the tails.
The visual loads as you reach this section.
Inspect the plotted values
x,y,series -6,0.0024726231566347743,Sigmoid -5.8,0.0030184163247084245,Sigmoid -5.6,0.003684239899435989,Sigmoid -5.4,0.004496273160941178,Sigmoid -5.2,0.005486298899450404,Sigmoid -5,0.0066928509242848554,Sigmoid -4.8,0.008162571153159897,Sigmoid -4.6,0.009951801866904324,Sigmoid -4.4,0.012128434984274237,Sigmoid -4.2,0.014774031693273055,Sigmoid -4,0.01798620996209156,Sigmoid -3.8,0.021881270936130476,Sigmoid -3.5999999999999996,0.026596993576865863,Sigmoid -3.4,0.032295464698450516,Sigmoid -3.1999999999999997,0.03916572279676437,Sigmoid -3,0.04742587317756678,Sigmoid -2.8,0.057324175898868755,Sigmoid -2.5999999999999996,0.06913842034334684,Sigmoid -2.4,0.08317269649392238,Sigmoid -2.1999999999999997,0.09975048911968518,Sigmoid -2,0.11920292202211755,Sigmoid -1.7999999999999998,0.14185106490048782,Sigmoid -1.5999999999999996,0.16798161486607557,Sigmoid -1.3999999999999995,0.19781611144141834,Sigmoid -1.1999999999999993,0.23147521650098246,Sigmoid -1,0.2689414213699951,Sigmoid -0.7999999999999998,0.31002551887238755,Sigmoid -0.5999999999999996,0.35434369377420466,Sigmoid -0.39999999999999947,0.40131233988754816,Sigmoid -0.1999999999999993,0.4501660026875223,Sigmoid 0,0.5,Sigmoid 0.20000000000000018,0.549833997312478,Sigmoid 0.40000000000000036,0.5986876601124521,Sigmoid 0.6000000000000005,0.6456563062257956,Sigmoid 0.8000000000000007,0.6899744811276126,Sigmoid 1,0.7310585786300049,Sigmoid 1.2000000000000002,0.7685247834990178,Sigmoid 1.4000000000000004,0.8021838885585818,Sigmoid 1.6000000000000005,0.8320183851339246,Sigmoid 1.8000000000000007,0.8581489350995123,Sigmoid 2,0.8807970779778823,Sigmoid 2.200000000000001,0.900249510880315,Sigmoid 2.4000000000000004,0.9168273035060777,Sigmoid 2.5999999999999996,0.9308615796566531,Sigmoid 2.8000000000000007,0.9426758241011313,Sigmoid 3,0.9525741268224334,Sigmoid 3.200000000000001,0.9608342772032357,Sigmoid 3.4000000000000004,0.9677045353015495,Sigmoid 3.6000000000000014,0.9734030064231343,Sigmoid 3.8000000000000007,0.9781187290638694,Sigmoid 4,0.9820137900379085,Sigmoid 4.200000000000001,0.9852259683067269,Sigmoid 4.4,0.9878715650157257,Sigmoid 4.600000000000001,0.9900481981330957,Sigmoid 4.800000000000001,0.9918374288468401,Sigmoid 5,0.9933071490757153,Sigmoid 5.200000000000001,0.9945137011005495,Sigmoid 5.4,0.9955037268390589,Sigmoid 5.600000000000001,0.9963157601005641,Sigmoid 5.800000000000001,0.9969815836752917,Sigmoid 6,0.9975273768433653,Sigmoid
The decision boundary is linear: logistic regression can only separate classes a straight hyperplane can divide (feature engineering or kernels are needed for nonlinear boundaries).
Work through the math
The model predicts:
Training minimizes binary cross-entropy over examples with labels :
A clean result falls out: the gradient with respect to the weights is
: prediction error times input, the same elegant form as a linear layer under cross-entropy. Why cross-entropy and not MSE? With the sigmoid, MSE is non-convex and its gradient vanishes when predictions are confidently wrong (the sigmoid saturates); cross-entropy is convex in the weights and keeps gradients strong, so it trains reliably. Add L2 regularization () to shrink weights and curb overfitting.
Key takeaway: Logistic regression is a linear model of the log-odds squashed by a sigmoid: trained with cross-entropy because MSE + sigmoid is non-convex with vanishing gradients exactly when the model is confidently wrong.
| Loss | With sigmoid output | Convex? | Gradient on confident-wrong errors |
|---|---|---|---|
| Cross-entropy (log loss) | Standard, well-behaved | Yes | Strong: keeps training moving |
| MSE | Non-standard, pathological | No | Vanishes: confident mistakes don't get corrected |
Read the implementation
1import numpy as np
2
3rng = np.random.default_rng(0)
4X = rng.normal(size=(200, 3))
5true_w = np.array([2.0, -1.0, 0.5])
6y = (X @ true_w + rng.normal(0, 0.5, 200) > 0).astype(float)
7
8def sigmoid(z): return 1 / (1 + np.exp(-z))
9
10w, b, lr = np.zeros(3), 0.0, 0.1
11for _ in range(2000):
12 p = sigmoid(X @ w + b)
13 grad_w = X.T @ (p - y) / len(y) # (p - y)·x: see Stage 3
14 grad_b = (p - y).mean()
15 w -= lr * grad_w
16 b -= lr * grad_b
17
18acc = ((sigmoid(X @ w + b) >= 0.5) == y).mean()
19print(f"accuracy {acc:.2f}, weights {w.round(2)}")Questions and trade-offs
- Conceptual: Why is it called "regression" if it does classification? (It regresses the log-odds (a linear model of log(p/(1−p))) then thresholds the resulting probability.)
- Implementation: Why train with cross-entropy instead of MSE? (With a sigmoid, MSE is non-convex and its gradient vanishes on confident-wrong predictions; cross-entropy is convex and keeps gradients healthy.)
- Applied: What does the decision boundary look like, and what's the limitation? (A linear hyperplane: it can't separate classes that aren't linearly separable without feature engineering or a kernel.)
- Systems-level: How do you extend logistic regression to more than two classes? (Softmax (multinomial) regression: generalize the sigmoid to a softmax over class logits with categorical cross-entropy.)
- Failure modes: How can you interpret the learned coefficients? (Each weight is the change in log-odds per unit change in that feature; sign and magnitude indicate direction and strength of influence, assuming scaled features.)
Check your understanding
Without looking: write the sigmoid model, the binary cross-entropy loss, and the weight gradient . Explain in one sentence why cross-entropy beats MSE here. Check against Stage 3.
Oliver Perrin
Machine Learning Engineer · Founder, LiminalML
LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.
Keep exploring the idea.
Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.
Related concepts
Classical ML
Bias-Variance Tradeoff
Separate systematic error from sensitivity to the training sample. Read an error decomposition without assuming every model follows a U-shaped curve.
Optimization
Gradient Descent (SGD, Momentum, Adam)
Follow updates across a loss surface, then connect learning rate, momentum, and Adam to the update equations.
Classical ML
Gradient Boosting & XGBoost
See one residual-fitting update, then connect the additive model to gradient-based boosting for other losses.