Logistic Regression
Connect a linear score to a sigmoid probability, a cross-entropy loss, and a decision threshold.
Jump to a section
Why this matters
Logistic regression turns a linear score into a probability with the sigmoid function. It is a classification model despite its name, and its parameters are commonly fitted by minimizing binary cross-entropy.
Distinguish the score, estimated probability, and decision threshold. Changing a threshold changes predictions without refitting the model. A linear decision boundary can still be useful, but it cannot express every relationship in the original features.
Build the mental model
The linear part draws a hyperplane through feature space. Points on one side score positive, the other negative. The sigmoid turns that raw score into a probability between 0 and 1, steeply near the boundary and saturating far from it. Predict class 1 when (i.e. when the linear score is nonnegative).
A linear score becomes a probability
The sigmoid surface crosses probability 0.5 where the linear score equals zero.
Exact inputs and equations appear with the figure.
Preparing the scientific figure…
Exact sigmoid probabilities for supplied coefficients. Fitting, calibration, and threshold selection are separate tasks.
Authored analytic exampleExperiment with related Studio inputs
Read the probability at zero, then compare equally sized changes near the center and in the tails.
Exact sigmoid values for a single logit. Choosing a decision threshold is separate from fitting or calibrating a classifier.
Inspect the related Studio values
x,y,series -6,0.0024726231566347743,Sigmoid -5.8,0.0030184163247084245,Sigmoid -5.6,0.003684239899435989,Sigmoid -5.4,0.004496273160941178,Sigmoid -5.2,0.005486298899450404,Sigmoid -5,0.0066928509242848554,Sigmoid -4.8,0.008162571153159897,Sigmoid -4.6,0.009951801866904324,Sigmoid -4.4,0.012128434984274237,Sigmoid -4.2,0.014774031693273055,Sigmoid -4,0.01798620996209156,Sigmoid -3.8,0.021881270936130476,Sigmoid -3.5999999999999996,0.026596993576865863,Sigmoid -3.4,0.032295464698450516,Sigmoid -3.1999999999999997,0.03916572279676437,Sigmoid -3,0.04742587317756678,Sigmoid -2.8,0.057324175898868755,Sigmoid -2.5999999999999996,0.06913842034334684,Sigmoid -2.4,0.08317269649392238,Sigmoid -2.1999999999999997,0.09975048911968518,Sigmoid -2,0.11920292202211755,Sigmoid -1.7999999999999998,0.14185106490048782,Sigmoid -1.5999999999999996,0.16798161486607557,Sigmoid -1.3999999999999995,0.19781611144141834,Sigmoid -1.1999999999999993,0.23147521650098246,Sigmoid -1,0.2689414213699951,Sigmoid -0.7999999999999998,0.31002551887238755,Sigmoid -0.5999999999999996,0.35434369377420466,Sigmoid -0.39999999999999947,0.40131233988754816,Sigmoid -0.1999999999999993,0.4501660026875223,Sigmoid 0,0.5,Sigmoid 0.20000000000000018,0.549833997312478,Sigmoid 0.40000000000000036,0.5986876601124521,Sigmoid 0.6000000000000005,0.6456563062257956,Sigmoid 0.8000000000000007,0.6899744811276126,Sigmoid 1,0.7310585786300049,Sigmoid 1.2000000000000002,0.7685247834990178,Sigmoid 1.4000000000000004,0.8021838885585818,Sigmoid 1.6000000000000005,0.8320183851339246,Sigmoid 1.8000000000000007,0.8581489350995123,Sigmoid 2,0.8807970779778823,Sigmoid 2.200000000000001,0.900249510880315,Sigmoid 2.4000000000000004,0.9168273035060777,Sigmoid 2.5999999999999996,0.9308615796566531,Sigmoid 2.8000000000000007,0.9426758241011313,Sigmoid 3,0.9525741268224334,Sigmoid 3.200000000000001,0.9608342772032357,Sigmoid 3.4000000000000004,0.9677045353015495,Sigmoid 3.6000000000000014,0.9734030064231343,Sigmoid 3.8000000000000007,0.9781187290638694,Sigmoid 4,0.9820137900379085,Sigmoid 4.200000000000001,0.9852259683067269,Sigmoid 4.4,0.9878715650157257,Sigmoid 4.600000000000001,0.9900481981330957,Sigmoid 4.800000000000001,0.9918374288468401,Sigmoid 5,0.9933071490757153,Sigmoid 5.200000000000001,0.9945137011005495,Sigmoid 5.4,0.9955037268390589,Sigmoid 5.600000000000001,0.9963157601005641,Sigmoid 5.800000000000001,0.9969815836752917,Sigmoid 6,0.9975273768433653,Sigmoid
The decision boundary is linear: it can fit overlapping, nonseparable data, but cannot represent a nonlinear boundary in the supplied feature space without a feature transformation or kernel.
Work through the math
The model predicts:
Training minimizes binary cross-entropy over examples with labels :
A clean result falls out: the gradient with respect to the weights is
: prediction error times input, the same elegant form as a linear layer under cross-entropy. Why cross-entropy and not MSE? With the sigmoid, MSE is non-convex and its gradient vanishes when predictions are confidently wrong (the sigmoid saturates); cross-entropy is convex in the linear model parameters and has logit gradient , avoiding the extra sigmoid factor. This does not guarantee rapid convergence; unregularized fitting on separable data can drive weight norms without bound. Add L2 regularization () to shrink weights and curb overfitting.
Key takeaway: Logistic regression is a linear model of the log-odds squashed by a sigmoid: trained with cross-entropy because MSE + sigmoid is non-convex with vanishing gradients exactly when the model is confidently wrong.
| Loss | With sigmoid output | Convex? | Gradient on confident-wrong errors |
|---|---|---|---|
| Cross-entropy (log loss) | Standard, well-behaved | Yes | Strong: keeps training moving |
| MSE | Extra sigmoid derivative in the gradient | Not generally | Can become very small on confident mistakes |
Read the implementation
1import numpy as np
2
3rng = np.random.default_rng(0)
4X = rng.normal(size=(200, 3))
5true_w = np.array([2.0, -1.0, 0.5])
6y = (X @ true_w + rng.normal(0, 0.5, 200) > 0).astype(float)
7
8def sigmoid(z): return 1 / (1 + np.exp(-z))
9
10w, b, lr = np.zeros(3), 0.0, 0.1
11for _ in range(2000):
12 p = sigmoid(X @ w + b)
13 grad_w = X.T @ (p - y) / len(y) # (p - y)·x: see Stage 3
14 grad_b = (p - y).mean()
15 w -= lr * grad_w
16 b -= lr * grad_b
17
18acc = ((sigmoid(X @ w + b) >= 0.5) == y).mean()
19print(f"accuracy {acc:.2f}, weights {w.round(2)}")Questions and trade-offs
- Conceptual: Why is it called "regression" if it does classification? (It regresses the log-odds (a linear model of log(p/(1−p))) then thresholds the resulting probability.)
- Implementation: Why train with cross-entropy instead of MSE? (With a sigmoid, MSE is non-convex and its gradient vanishes on confident-wrong predictions; cross-entropy is convex and keeps gradients healthy.)
- Applied: What does the decision boundary look like, and what's the limitation? (A linear hyperplane: it cannot express a nonlinear boundary in the supplied features, but can still fit overlapping, nonseparable classes.)
- Systems-level: How do you extend logistic regression to more than two classes? (Softmax (multinomial) regression: generalize the sigmoid to a softmax over class logits with categorical cross-entropy.)
- Failure modes: How can you interpret the learned coefficients? (Each weight is the change in log-odds per unit change in that feature; holding other features fixed. Coefficients are associations, not causal effects; units and feature correlations affect interpretation.)
Check your understanding
Without looking: write the sigmoid model, the binary cross-entropy loss, and the weight gradient . Explain in one sentence why cross-entropy beats MSE here. Check against Stage 3.
Sources and further reading
- scikit-learn, Logistic regression: linear logits, classification loss, and regularization.
Oliver Perrin
Machine Learning Engineer · Founder, LiminalML
LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.
Keep exploring the idea.
Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.
Related concepts
Classical ML
Bias-Variance Tradeoff
Separate systematic error from sensitivity to the training sample. Read an error decomposition without assuming every model follows a U-shaped curve.
Optimization
Gradient Descent (SGD, Momentum, Adam)
Follow updates across a loss surface, then connect learning rate, momentum, and Adam to the update equations.
Classical ML
Gradient Boosting & XGBoost
See one residual-fitting update, then connect the additive model to gradient-based boosting for other losses.