Linear & Logistic Regression
You'll learn to
- -Understand linear regression as fitting a straight-line relationship for numeric prediction
- -Derive and implement the closed-form least-squares solution, and check it by hand on five points
- -Understand logistic regression as linear regression squashed through a sigmoid for classification
- -Read the sign and size of a weight: what it says about the relationship, and where a prediction crosses 50%
Linear regression is often the very first ML algorithm anyone learns, and for good reason. It is simple, interpretable, and still genuinely useful in production for the right kind of problem.
An Analogy: A Trend Line and a Dimmer Switch
A trend line through a scatter of points is linear regression: for every extra unit of the input, the output moves by a roughly fixed amount, such as every extra 1,000 sq ft adding about the same to a house's price. Logistic regression adds a dimmer switch on the end. Instead of the raw output of the line, which can be any number, the switch eases it into a smooth value between 0 (off) and 1 (on) that you can read as a probability.
- -The weight w is the slope of the trend line: how much the output moves per unit of input.
- -The bias b is where the line starts when the input is zero.
- -The sigmoid is the dimmer switch: it turns any score into a probability between 0 and 1.
Linear Regression: Fitting a Line (or Plane)
Given examples of inputs and a numeric output, linear regression finds the best-fitting straight-line relationship between them, or with multiple inputs, the best-fitting flat plane through higher-dimensional space. "Best-fitting" means the one that minimizes a loss function, usually squared error, across all the training examples.
The Closed-Form Solution
Unlike most ML models, simple linear regression does not need gradient descent at all. Minimizing squared error has an exact, closed-form answer you can compute directly from the data in one pass.
Checking the Formula by Hand
The five houses in the code below have sizes 1.0, 1.5, 2.0, 2.5, 3.0 (in thousands of sq ft) and prices 200, 250, 310, 370, 410 (in thousands of dollars). Work it through:
- -The means are x-bar = 2.0 and y-bar = 308.
- -The top of the slope formula is the sum of (x - x-bar)(y - y-bar): 108 + 29 + 0 + 31 + 102 = 270.
- -The bottom is the sum of (x - x-bar) squared: 1 + 0.25 + 0 + 0.25 + 1 = 2.5.
- -So w = 270 / 2.5 = 108, and b = 308 - 108 x 2 = 92. The line is price = 108 x size + 92.
The line predicts 200, 254, 308, 362 and 416 for the five houses, so the errors (called residuals) are 0, -4, 2, 8 and -6. Squaring and adding them gives 0 + 16 + 4 + 64 + 36 = 120. No other straight line has a smaller sum of squared errors, which is exactly what least squares means. A house of 2.2 thousand sq ft is predicted at 108 x 2.2 + 92 = 329.6, about $330k.
Logistic Regression: The Classification Cousin
Despite the name, logistic regression is used for classification, not regression. It takes the same underlying idea, a weighted combination of inputs, and squashes the result through the sigmoid function, which always outputs a number between 0 and 1 that reads as a probability.
The sigmoid is easy to get a feel for with three values. A score of 0 gives exactly 0.5, the fence. A score of +2 gives about 0.88, and -2 gives about 0.12. Large positive scores approach 1 and large negative scores approach 0, so any score becomes a valid probability.
Read the churn model through its weight. With w = 0.15 and b = -1.2, the probability is 0.26 at 1 month, 0.43 at 6, 0.65 at 12 and 0.92 at 24. The positive sign says the longer a customer is inactive, the more likely they are to churn. The model crosses the 50% fence where 0.15 x months - 1.2 = 0, which is at 8 months. The sign tells you the direction, and the size tells you how quickly the probability moves.
- -Linear regression predicts a continuous number: price, temperature, delivery time.
- -Logistic regression predicts a probability, used for binary classification: churn or no churn, spam or not spam.
Both models are "linear" in the sense that they combine inputs by weighting and summing them, with no interactions or curves unless you explicitly engineer them in. That makes them fast, interpretable, and a strong first baseline, even when a more complex model eventually outperforms them.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Build all three from scratch, then apply logistic regression to a real diagnostic dataset: Level 5 (Fitting the Line), Level 6 (Descending the Bowl), Level 7 (The Sigmoid Gate), and Level 36 (When Missing One Matters More Than a False Alarm, on real breast-cancer data) in AI Lab.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!