Editorial
White Paper
The New Publishing Standard in the AI Era and What It Replaces
The Gist
- Randomize markets. A/B testing can't reach TV, audio, out-of-home or most streaming video, so geo experiments randomize markets instead of people to close the measurement gap.
- Stretch small samples. Netflix built an internal tool called Quasimodo to toggle ads on and off across a limited number of media markets, turning one campaign into several statistical reads.
- Lead the hard calls. Holdout size, protected test windows and test duration are leadership decisions, not analyst defaults — teams that own them build an archive that guides future budgets.
Automotive engineers test cold-start performance by sending test fleets into different climates and comparing results against how vehicles normally perform in each place.
Marketers face a similar problem with much of their media budget. Nobody can choose which person sees a billboard or withhold a TV spot from half the households in a market. That limit is getting fresh attention after Karan Dhir, a marketing measurement leader, published a breakdown on Substack of how Netflix measures the marketing it can't A/B test.
As CFOs press CMOs for proof of incremental revenue from data-driven initiatives, sophisticated market-level experiments are becoming the best way to validate channels where the most investment and the least user-level data sit. As a result, organizations that run geo testing are discovering the benefits from the validation, as well as the risks. What follows covers the blind spot A/B testing leaves across major channels, why incrementality measurement is gaining ground, what Netflix's engineering material reveals about running geo experiments well, and the lessons, vendor questions and starter tests that build a practice.
Why A/B Testing Can't Measure Most Marketing Channels
For about two decades, A/B testing has trained marketers to expect clean answers. Split the traffic, change one element, wait for significance and ship the winner. That discipline still earns its place in the martech stack, and its results show how hard customer behavior is to move. An Optimizely analysis of more than 127,000 experiments found only 12% produced a statistically significant improvement on the primary metric. Revenue-focused tests delivered an average incremental lift of 0.4% in digital revenue once results were applied and built upon.
Those small, compounding gains depend on a condition many channels can't meet. A/B testing works when a brand controls who sees what, and that control ends at the edge of owned digital properties. Sponsorships, audio, out-of-home, linear TV, much of streaming TV and a big share of upper-funnel video resist individual-level randomization, Dhir notes, and at most established brands those channels carry the majority of spend.
Customer experience is part of this, too. Customers pass a billboard, hear a podcast ad and search for the brand days later. Click-based reports often credit that final search, so earlier touchpoints look weaker than they are and budgets drift toward channels that intercept existing demand.
Marketers have noticed that gap. eMarketer reports that 52% of US brand and agency marketers already use incrementality testing, and 36.2% plan to invest more in it over the next year. That shift changes workflows. Media planners schedule holdouts alongside campaign flights, analysts lock in success metrics before launch and finance weighs in on how much delivery a test can give up.
Related Article: The 5-Step Martech Reality Check for Marketing Leaders
Why Can't A/B Testing Measure Most Marketing Channels?
A/B testing requires individual-level control that ends at owned digital properties, leaving TV, audio, out-of-home and sponsorships — often the majority of spend — unmeasured by traditional split tests.
Why Market-Level Experiments Are Gaining Ground
Geo experimentation's rise starts with a trust problem. The IAB State of Data 2026 report, based on more than 400 senior planning and analytics decision-makers at US brands and agencies, found 60% to 75% of buy-side users believe advanced measurement falls short on rigor, timeliness, trust and efficiency.
Brand leaders feel the pressure most. A Gartner survey found 84% of companies are stuck in a brand "doom loop," where underfunded measurement leads to unclear impact, rising skepticism and tighter budgets. Those companies were half as likely to exceed growth targets as firms able to evaluate brand value.
Confidence is shifting toward experiments. A January 2026 Haus survey reported by eMarketer found 60% of US senior decision-makers trust independent incrementality testing most, ahead of media mix modeling at 40% and in-platform reporting at 37%. Haus sells incrementality testing, so treat the exact figures as vendor research, though the direction matches the IAB findings.
The method has a long track record. Google researchers Jon Vaver and Jim Koehler described randomized geo experiments for advertising in 2011, and a 2017 Google paper added a time-based regression framework. The best-known demonstration came from eBay. Field experiments published in Econometrica, including paid search shutoffs in selected designated market areas, found brand-keyword ads had no measurable short-term benefit and paid search returns were a fraction of non-experimental estimates. As Dhir points out, eBay learned what attributed reporting couldn't show only because someone was willing to hold markets out.
The table below compares the two testing approaches across the dimensions that shape a measurement plan.
A/B Tests vs. Geo Experiments
A/B tests and geo experiments answer related questions at different scales. A/B tests optimize experiences a brand controls, and geo experiments validate channels where individual-level randomization isn't possible.
| Dimension | A/B Test | Geo Experiment |
|---|---|---|
| Unit of randomization | Individual user, session or device | Markets such as DMAs or TV media markets |
| Typical channels | Websites, apps, email and some paid social | TV, out-of-home, audio, sponsorships, streaming and paid search |
| Sample size | Thousands to millions of users | Often a few dozen markets or fewer |
| Data required | User-level exposure and conversion data | Market-level outcomes and spend, with no user tracking |
| Primary risk | Weak setup, early stopping and loose metrics | Spillover across borders, uneven markets and low statistical power |
| Business cost | Low, limited to a slice of traffic | Higher, since holdout markets give up media delivery |
| Best use | Optimizing experiences, offers and creative | Validating channel investment and model estimates |
The comparison shows how the methods complement each other. A/B tests give fast feedback on experiences, and geo experiments supply channel-level causal reads that validate media mix models. Geo tests need only market-level outcomes and spend, so they hold up as user tracking declines. Their trade-offs, small samples and lost media delivery, are where Netflix's approach becomes instructive.
The sampling questions start earlier than most teams expect.
Michael Frohlich, developer and full stack engineer at Knowroaming.com, sees geo testing as an expansion of A/B testing that carries its own trade-off.
"Incorporating geo experience into your A/B testing can help take your experiments to the next level," he said. "You suddenly have the capability to test a much wider geographic range in your samples."
Selection bias is big concern in this arena, he added. "Since you're moving from testing subjects to testing whole geographic regions, you're obviously not getting user-specific results," he said. "The people in one region may have different patterns and demand than those in another geographic group. They may both use your product, but not at the same times."
While randomization in testing is a good and necessary thing in many regards, Frohlich said if it's too random or not random enough, you run into problems and end up with skewed results.
"It's great that you're targeting an entire geographic region and widening your audience," he said, "but you're not always getting the best data at the user level. You have to have protocols in place to account for any regional discrepancies."
The protocols Frohlich describes are exactly what the next section covers. Regional differences in timing, demand and product usage can masquerade as campaign lift, so the balancing work has to happen before a single ad runs.
Why Are Marketers Turning to Market-Level Experiments?
Trust in existing measurement is low — 60–75% of buy-side marketers doubt advanced measurement's rigor — while independent incrementality testing now earns the highest trust score among available methods.
How Netflix Runs Geo Experiments at Scale
At its core, a geo experiment compares markets that received a campaign against markets that didn't, adjusted for how those markets tracked each other beforehand. Netflix randomizes at the smallest level where it can buy media, which for TV in most countries is the media market, usually a cluster of nearby cities.
That creates a small-sample problem, since a country holds few media markets and they differ widely. Netflix engineers use a UK example. If London, with its strong appetite for sci-fi, lands in the treatment group for a sci-fi series campaign, a simple comparison overstates the marketing's impact. The team re-randomizes until test cells balance on key variables, then uses difference-in-differences comparisons that normalize results against each market's pre-campaign baseline.
As test volume grew, Netflix began building Quasimodo inside its experimentation platform to automate the mechanical work. For low-sample tests with historical data, the team switches advertising on and off repeatedly across markets, and each toggle offers another read that Quasimodo estimates with a dynamic linear model, comparable to Google's CausalImpact package but able to handle multiple interventions across regions. Dhir sums up the value. A takedown becomes a second measurement, a relaunch in fresh markets a third, so one campaign buys statistical power the market count can't supply.
Eight Design Choices That Separate Rigorous Geo Tests From Noise
The following table highlights the most important lessons, actions and strategic considerations emerging from Netflix's published geo experiment methods and Dhir's argument that the most consequential choices belong to leadership.
Those details translate into practical guidance. The lessons below combine Netflix's published methods with Dhir's argument that the most consequential geo testing choices are leadership decisions.
| Key Area | What Happened | Why It Matters | Recommended Action |
|---|---|---|---|
| Unit of randomization | Netflix randomizes at the smallest unit it can buy, typically the media market. | Results only translate into action when they match the level planners actually buy at. | Set test cells to media-buying boundaries such as DMAs, not arbitrary geographic splits. |
| Market balance | An outlier market like London, with unusually strong sci-fi demand, can skew a simple comparison. | One unbalanced market can overstate or understate a campaign's true effect. | Re-randomize test cells against pre-period baselines before launch. |
| Test volume | Netflix treats each launch, takedown and relaunch as a separate statistical read. | Most countries have too few media markets to power a single clean test. | Sequence campaign flights so takedowns and relaunches generate additional reads. |
| Design fit | Product and title launches have no pre-period history for a time-series model to compare against. | Forcing a time-series design onto missing history produces unreliable results. | Switch to customer-level controls when a pre-period baseline doesn't exist. |
| Holdout size | Advertising effects are small relative to normal sales swings, so weak holdouts yield ambiguous reads. | Holdouts shrunk to avoid internal friction let each side interpret results through existing beliefs. | Have leadership set holdout size and duration, not analysts by default. |
| Test window integrity | Spillover, commuting patterns and national promotions can contaminate test markets mid-flight. | A contaminated window invalidates the comparison between treatment and control markets. | Assign a single owner with real authority over the experiment calendar. |
| Duration vs. conclusions | A six-week holdout measures only a six-week effect. | Extending a short-term finding into a permanent budget cut risks missing carryover and brand effects. | Bound every conclusion to the actual duration the test ran. |
| Program continuity | A single geo test is, as Dhir puts it, an anecdote with confidence intervals. | One-off tests can't calibrate media mix models or build durable organizational confidence. | Run rolling retests and keep a searchable archive that feeds model calibration. |
The first four lessons are technical safeguards an analytics team can own. The last four trade media delivery against measurement certainty, so they need leadership. Left to analysts by default, those choices often produce ambiguous results. Made explicit, they produce answers finance can act on.
What Does Netflix's Geo Experiment Process Teach Marketers?
Netflix randomizes at the smallest buyable media unit, rebalances test cells against pre-campaign baselines, and reuses campaign launches, takedowns and relaunches as multiple statistical reads via its Quasimodo tool.
FAQ: Geo Experiments and Marketing Measurement
Editor's note: These questions address how geo experiments differ from A/B testing and what's required to run one credibly.
Netflix randomizes at the media-market level, rebalances test cells against pre-campaign baselines, and uses an internal tool called Quasimodo to treat campaign launches, takedowns and relaunches as separate statistical reads.A geo experiment randomizes markets — such as media markets or DMAs — rather than individual users, comparing campaign and holdout markets to measure channels A/B testing can't reach.Marketing leadership, not analysts by default — these choices trade media delivery against measurement certainty and directly affect whether results are strong enough for finance to act on.A/B testing requires individual-level control over who sees an ad. TV, audio, out-of-home and much of streaming video can't be split at the person level, so they resist randomized testing.
What Marketers Should Do Next
Dhir proposes a standard worth bringing to any leadership team. Every channel receiving eight figures of annual investment should have a market-level causal read less than 24 months old, built on a pre-registered analysis, adequate power and a protected test window. Channels that fall short should carry an uncertainty label in planning. Smaller brands can scale the threshold to their budgets.
Expect friction. In retail media, top barriers to incrementality measurement include doubts about result accuracy at 44%, applying incrementality across ad types and retailers at 43% and limited tools at 41%, according to Skai and Path to Purchase Institute data reported by eMarketer. Setup discipline eases the accuracy worry. A newer Optimizely analysis of 173,000 experiments found soundly configured tests won 58% of the time versus 35% for poorly configured ones, at a 90% significance threshold.
Practitioners who manage regional budgets every day see limits worth weighing before a program launches.
Michael Robichaud, founder of Chicago-based Strategized.io, raises two concerns drawn from twenty years running Google Ads at the account level, where region is a budget variable rather than a randomized unit.
"Geography is no longer a clean test subject," Robichaud said. "Personalization and localization move the result underneath you. I verify rank claims in a rendered browser instead of an API parse for exactly that reason: a neutral check put a page I track at #9 in the US where a personalized screenshot showed #2. Same query, same day. If one result set swings seven positions on personalization alone, a regional split has to prove it isolated geography and not the audience that happens to live there."
The second concern is quieter, according to Robichaud. Attribution behaves differently when the journey starts inside an AI assistant rather than a results page, he added.
"An assistant's answer is not geo-partitioned the way a SERP is, so part of your test population sees something the regional split never touched," Robichaud said. "That contamination does not announce itself. It makes the lift read smaller or larger than it was, and you write it up either way."
Netflix has the volume to absorb both. A mid-size advertiser does not, which is the sample-size problem geo testing was built to solve arriving through a different door.
Robichaud's warning sharpens the design work. Document what makes treatment and control markets comparable in audience terms rather than population alone, and account for demand that begins inside an AI assistant. Mid-size advertisers gain the most by testing their largest channels first, running longer windows and treating early reads as directional.
What Questions Should Marketers Ask a Geo Testing Vendor?
Before signing with a measurement vendor or platform, marketing leaders should ask three pointed questions.
- How are test markets selected and balanced? Look for re-randomization or matched-market design and clarity on which variables drive balance.
- What power analysis runs before launch? The vendor should translate your minimum effect worth detecting into a specific holdout size and duration.
- How is contamination handled? Ask how the tool treats spillover between markets and overlapping national campaigns.
Those answers show whether a tool supports an ongoing program or a one-time study, and whether its outputs can calibrate a media mix model later. Teams also don't need a Netflix-sized budget to start. Google has released open-source code for its geo experiment methods, and Meta maintains the open-source GeoLift package for synthetic control analysis.
A practical first experiment is a paid search holdout across matched markets, since that spend is easy to geo-target and results arrive quickly. Next, try a launch-and-takedown sequence for a regional out-of-home or audio flight to read the effect twice. Before either test, write down the success metric, the smallest effect worth detecting and the decision the result will trigger.
Geo experimentation has been published, replicated and industrialized by leading measurement teams, so what separates organizations now is commitment. Marketers who build the habit will enter each planning cycle knowing which channel estimates were tested and which remain assumptions, and that clarity earns budget credibility with the C-suite.
Learn how you can join our contributor community.
