Offline Recommender Evaluation With Human Raters: When Clicks Are Not Enough

The short answer: Offline recommender evaluation with human raters means showing trained raters a user context and recommendations from one or more models, asking them to judge relevance, quality and diversity against a rubric, and comparing models blind and side by side. It catches problems engagement metrics reward, such as clickbait, repetition and poor fit for new users, and helps teams decide which changes deserve live tests. Many teams source raters through specialist recommendation data labeling programs.

Why clicks are not enough

Recommenders are usually optimized for engagement: clicks, views, watch time and purchases. These signals are abundant and useful, but they reward whatever grabs attention. Sensational thumbnails, repeated items and narrow feeds can raise short-term engagement while users grow tired or lose trust. Engagement also says little about new items and new users, which have little history.

Human judgment adds what behavior misses: whether a recommendation actually fits the user, whether it is good quality and whether the set feels varied and fresh. Offline evaluation brings that judgment into the development cycle before changes go live.

Give raters clear user context

Raters cannot judge relevance without knowing who the recommendations are for. Each task should present a user context: a profile, recent history, a stated goal or a scenario such as shopping for a gift or looking for something new to watch. The context should be realistic and drawn from real usage patterns, with personal data removed.

Clear context is what separates relevance judgments from personal taste. When raters disagree, it is often because the context left room for interpretation, and improving it raises agreement more than retraining raters does.

Write a rubric for relevance, quality and diversity

Rubrics should define graded relevance levels, such as excellent fit, good fit, weak fit and wrong, with examples for each. They should also cover item quality, such as misleading titles or poor listings, and set-level qualities such as repetition, staleness and variety. Rules for sensitive items, such as age-restricted or controversial content, belong in the rubric too.

Raters should give short reasons for their judgments. Reasons make calibration easier and help model teams understand why a recommendation failed.

Compare models blind and side by side

The most useful offline test is often side by side: raters see two recommendation lists for the same user context, produced by different models, without knowing which is which, and choose the better list with reasons. Positions are randomized so neither model benefits from always appearing first. Win rates by user type and item category show where a new model helps and where it hurts.

Side-by-side tests are more sensitive than absolute scores for small changes, and they mirror the decision teams actually face: ship the new model or not. See model evaluation for comparison methods used across AI models.

Build versioned evaluation sets

Evaluation sets should cover the situations that matter: heavy users and new users, popular and niche items, different markets and categories, and cold-start cases where items or users have little history. Sets should be versioned and reused, so results stay comparable across model versions.

Refresh sets periodically so they reflect current catalogs and behavior, but keep a stable core for long-term comparisons. Separate evaluation contexts from any data used to train or tune models.

Testing cold start and new items

New items and new users are where recommenders struggle most. Offline sets should include contexts with little history and recommendations that feature new items, so raters can judge whether the system places them well. Item attributes labeled by humans help here, because they let models understand new items before behavior data arrives.

See marketplace catalog management for how item attributes are maintained, and how attribute quality affects new-item recommendations.

Judging diversity, novelty and fairness

Engagement-optimized feeds can narrow over time and concentrate exposure on a few popular creators or sellers. Raters can judge whether a feed feels varied, whether it introduces something new and whether smaller creators get fair exposure. These judgments guide diversity rules and ranking adjustments.

Policy issues found during evaluation, such as unsafe content, should go to trust and safety teams. See trust and safety for policy enforcement.

Using offline results with A/B tests

Offline human evaluation does not replace live experiments; it decides which changes deserve them. Changes that lose clearly offline can be dropped early, saving traffic and time. Changes that win offline go to A/B tests, where engagement and business metrics confirm the effect.

Comparing offline and online results over time shows how well the offline setup predicts real outcomes, and helps teams refine contexts and rubrics.

Rater selection and calibration

Raters should know the domain and the market: fashion, music, food or travel taste is cultural. Screen raters with sample tasks, calibrate them on gold examples and measure agreement over time. Raters who drift are recalibrated or reassigned.

Keep raters blind to which model produced which list, and rotate tasks so nobody learns to recognize a model’s style.

What to report from each evaluation

Report win rates overall and by user type, item category and market; relevance score distributions; diversity and quality findings; agreement between raters; and example failures with reasons. Example failures are often the most useful part, because they show model teams exactly what went wrong.

Keep a history of results across model versions so progress is visible over time.

Running a human evaluation program

Start with one surface, such as home feed or related items, build a rubric and an evaluation set, calibrate raters and run a side-by-side test on a real model change. Expand to other surfaces and markets once results prove useful.

Our recommendation data labeling outsourcing page explains how vetted Philippine raters judge relevance, similarity and feed quality, and how to detect reward hacking covers a related evaluation problem in the same cluster.