Skip to content
Join / Sign In
GenAI Learn/The Classical ML Toolbox
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Linear & Logistic Regression

7 min read

You'll learn to

  • -Understand linear regression as fitting a straight-line relationship for numeric prediction
  • -Understand logistic regression as linear regression adapted for classification
  • -Derive and implement the closed-form least-squares solution for simple linear regression

Linear regression is often the very first ML algorithm anyone learns, and for good reason. It is simple, interpretable, and still genuinely useful in production for the right kind of problem.

Linear Regression: Fitting a Line (or Plane)

Given examples of inputs and a numeric output, linear regression finds the best-fitting straight-line relationship between them, or with multiple inputs, the best-fitting flat plane through higher-dimensional space. "Best-fitting" means the one that minimizes a loss function, usually squared error, across all the training examples.

Simple Linear Regression
ŷ = wx + b
One input feature x, one weight w (the slope), and one bias b (the intercept). Multiple linear regression is the same idea with a vector of weights, one per feature.

The Closed-Form Solution

Unlike most ML models, simple linear regression does not need gradient descent at all. Minimizing squared error has an exact, closed-form answer you can compute directly from the data in one pass.

Least-Squares Slope and Intercept
w = Σ(xᵢ−x̄)(yᵢ−ȳ) / Σ(xᵢ−x̄)² , b = ȳ − wx̄
x̄ and ȳ are the means of x and y. The slope w is the covariance of x and y divided by the variance of x. The intercept b makes the line pass through the point (x̄, ȳ).

Logistic Regression: The Classification Cousin

Despite the name, logistic regression is used for classification, not regression. It takes the same underlying idea, a weighted combination of inputs, and squashes the result through the sigmoid function, which always outputs a number between 0 and 1 that reads as a probability.

Logistic Regression
ŷ = σ(wx + b) = 1 / (1 + e^−(wx+b))
The same linear combination wx + b as before, passed through the sigmoid function σ to compress it into a valid probability. "Will this customer churn?" becomes a number like 0.73.
Closed-form linear regression, plus a logistic regression prediction function
  • -Linear regression predicts a continuous number: price, temperature, delivery time.
  • -Logistic regression predicts a probability, used for binary classification: churn or no churn, spam or not spam.

Both models are "linear" in the sense that they combine inputs by weighting and summing them, with no interactions or curves unless you explicitly engineer them in. That makes them fast, interpretable, and a strong first baseline, even when a more complex model eventually outperforms them.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Optimization Intuition, Without the CalculusDecision Trees & Ensembles
ScaleDojo Logo
Initializing ScaleDojo