Distance-Based & Margin Methods
You'll learn to
- -Understand k-nearest neighbors as prediction by similarity to stored examples
- -Implement a k-nearest neighbors classifier from scratch
- -Understand the core idea behind support vector machines: maximizing the margin between classes
This chapter covers two classic approaches that both lean on the vector-and-distance intuition from the Math module. K-nearest neighbors predicts by looking at what similar examples did, and support vector machines find the boundary that separates classes as confidently as possible.
K-Nearest Neighbors: Predict Like Your Neighbors
K-nearest neighbors, or KNN, is almost embarrassingly simple. To predict for a new example, find the K most similar examples already seen in training, using vector distance, and let them vote. For classification, take the majority label among those K neighbors. For regression, average their values. There is no real training step at all beyond storing the data. All the work happens at prediction time.
KNN's simplicity is also its weakness. It must compare a new example against potentially every stored example to make a single prediction, which becomes slow as the dataset grows large. It works best on smaller datasets, or paired with faster approximate-search techniques, exactly the problem vector databases in Phase 2 are built to solve at scale.
KNN also has a hidden dependency worth flagging now and expanding on soon: because it relies entirely on distance, features on wildly different scales can quietly dominate the result. The Data Pipeline module covers feature scaling in depth, and its demo reuses this exact age-and-income scenario to show the effect directly.
Support Vector Machines: Maximize the Margin
A support vector machine, or SVM, also separates classes, but instead of just drawing any boundary that works, it specifically looks for the boundary with the widest possible margin: the largest buffer zone between the closest examples of each class. Intuitively, a wider margin means the boundary is less likely to misclassify a new example that lands close to the edge.
- -KNN needs no real training, is comparison-heavy at prediction time, and stays intuitive and interpretable for small datasets.
- -SVM training finds an optimal, maximally confident separating boundary, and can handle non-linear boundaries using a technique called the kernel trick, which projects data into a higher-dimensional space where a straight-line boundary becomes possible.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.