Framing the Problem: Objectives & Labels

Every supervised model learns to predict a label, and the label is a choice. Choose it well and the model drifts toward what you want. Choose it badly and the model becomes very good at the wrong thing. Much of ML system design is deciding what to predict and how to get examples of it.

A teacher who pays students for every page they write will get very long essays, not good ones. The students are not being difficult. They are optimizing exactly what the rule rewards. A model does the same with its label. If the label is "the user clicked", the model learns to produce clicks, including the cheap kind a misleading headline earns.

• -Explicit feedback: ratings, thumbs, reports. Clean and meaningful, but rare, because most users never give any.
• -Implicit feedback: clicks, watch time, purchases, skips, saves. Plentiful, but it measures behavior, which is not always satisfaction.
• -Human annotation: trained raters label examples. Accurate and expensive, so it is used to build evaluation sets and to seed training data.
• -Programmatic labels: rules or an existing system produce labels automatically. Cheap and noisy, so spot-check them.
• -Delayed ground truth: the real outcome arrives later, such as a chargeback weeks after a payment. Reliable, but late.

The YouTube recommendation paper from 2016 is a good example from industry. Its ranking model was trained to predict expected watch time, not the chance of a click, because a click objective tends to reward misleading thumbnails and titles. The principle is general: pick a label that is closer to the real goal than the easiest signal to collect.

The block below has two parts. The first part shows label delay, and you will read it in the next section. The second part scores four articles on click rate and on a stricter signal, a click followed by 30 seconds of reading. Press Run and compare the two rankings at the bottom.

The click rate puts the clickbait headline first, with 12.0%. Its satisfied-click rate is 1.4%, the lowest of the four. Rank by long reads instead and the useful guide moves to first, and the clickbait falls to last. The label you choose decides which item the model learns to push.

Many outcomes take time to appear. A purchase may come days after the ad. A fraud chargeback can take weeks. If you train on a snapshot taken too soon, many true positives are still missing, so they look like negatives. In the first part of the output, only 39% of eventual purchases are visible after one day, 78% after three days, and 97% after a week. A label read after one day would say the purchase rate is 1.57% when the true rate is 4.0%, and a model trained on it would learn to under-predict.

There are three common responses. Wait until labels mature before using an example for training, even if that means the model learns from slightly older data. Model the delay explicitly, treating the examples whose outcome is still pending differently. Or choose an earlier proxy label that arrives fast, and check it against the slow true label from time to time.

Google's guide recommends a first objective that is simple, observable, and attributable, and says not to overthink it. Observable means you can measure it from logs. Attributable means you can tell the model's contribution from everything else going on. A metric like "long-term customer happiness" is the true goal, but you cannot compute it per example, so you start with a proxy that you can, and refine as you learn what the proxy misses.