Bias & Fairness in ML Systems

A model trained on biased data will learn and often amplify that bias. This is not a rare edge case. It is the default outcome unless someone specifically works to prevent it. This chapter is about understanding how that happens, and why "just remove the bias" is harder than it sounds.

A resume-screening model trained on a company's historical hiring decisions will learn whatever patterns, including unfair ones, were present in those historical decisions, even if no one involved in building the model intended any unfairness at all. The data itself is a record of the world, and its existing inequities, as it actually was, not as it should be, and a model trained on it will faithfully reproduce that pattern unless something is deliberately done about it.

Here is a genuinely difficult, often underappreciated fact. There are multiple reasonable, competing mathematical definitions of fairness, and it is mathematically proven that you cannot simultaneously satisfy all of them in most real situations. For example, "the model should have equal accuracy across groups" and "the model should have equal false-positive rates across groups" can directly conflict with each other. Satisfying one can require violating the other. Choosing which definition of fairness matters most for a given system is a genuine ethical and product decision, not something a purely technical fix resolves once and for all.

• -Audit training data for representation gaps before training, not after deployment.
• -Measure model performance broken out by relevant subgroups, not just in aggregate.
• -Explicitly decide which fairness definition matters most for this specific system, and document that decision.
• -Treat fairness as an ongoing monitoring concern, not a one-time checkbox at launch.