Feature Scaling & Encoding Categorical Data
You'll learn to
- -Understand why unscaled features silently break distance-based and gradient-based models
- -Implement min-max scaling and z-score standardization from scratch
- -Implement one-hot encoding for categorical features
This chapter covers a step that is easy to skip and expensive to skip: getting every feature onto a comparable scale, and turning non-numeric, categorical, features into numbers a model can actually use. Skipping it does not usually cause an error. It causes a model that silently performs far worse than it should, which is a more dangerous failure mode.
Why Scale Matters: One Feature Can Dominate
Recall the k-nearest neighbors chapter's customer-classification example: age and income. Income ranges over tens of thousands, while age ranges over maybe 70 years. Euclidean distance squares each difference before summing, so a $10,000 income difference contributes 100,000,000 to the squared distance, while a 10-year age difference contributes only 100. Age becomes almost irrelevant to every distance calculation purely because of its units, regardless of how predictive it actually is. The same problem distorts gradient descent, since a feature with a much larger scale produces a much larger gradient, which can make training unstable or force an awkward learning-rate compromise across features with wildly different scales.
See it directly below. The query point is the same in both views. Only the distance calculation changes.
Same Data, Same Query Point, Different Neighbors
The gray diamond is a 46-year-old earning $42k. Which 3 points count as "nearest" depends entirely on whether the features are scaled first.
Unscaled KNN (k=3) predicts: label 0
Income differences (tens of thousands) swamp age differences (tens) in raw distance, so the neighbors are chosen almost entirely by income.
Min-Max Scaling
Z-Score Standardization
A subtle but important rule: fit the scaler, meaning compute the min, max, mean, or standard deviation, using only the training set, then apply that same fitted transformation to the validation and test sets. Never recompute those numbers on validation or test data. Doing otherwise is a form of the data leakage this course already flagged in the train, validation, and test split chapter.
Encoding Categorical Features
Most ML algorithms, everything covered in the Classical ML Toolbox and Neural Networks modules, only accept numbers as input. A feature like city, with values such as Chicago, Denver, or Miami, has to become numeric before it can be used at all.
- -One-hot encoding creates one new binary column per category: "is_chicago," "is_denver," "is_miami," so no category is treated as numerically bigger or closer to another. This is the standard choice for nominal, unordered, categories.
- -Label encoding assigns each category a single integer, such as Chicago=0, Denver=1, Miami=2. It is compact, but dangerous for nominal categories, since it implies an ordering and distance that a distance-based or linear model will take literally, even though Miami is not actually "further" from Chicago than Denver is. Reserve it for genuinely ordinal categories, like small=0, medium=1, large=2, where that ordering is real.
In practice, scikit-learn's StandardScaler and MinMaxScaler implement the two scaling formulas above, fit on train, transform on train, validation, and test. Its OneHotEncoder, or pandas' pd.get_dummies, implements one-hot encoding. All in a couple of lines, instead of the loops above.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.