EVERSANA selects GrowthLoop to power its AI-driven marketing: Read the case study

New research raises questions about historical LiveRamp match quality and limitations

Key takeaways:

  • A high match rate can still miss most of your customers. Match rate only counts how many records got linked to an ID. Match quality counts how many of those links are the right person.

  • Poor matches hurt more than ad spend, and your dashboards won’t show it. In the historical data studied, LiveRamp’s matched identities agreed with submitted age and gender 21.8% of the time, against a 10% random baseline.

  • Ask for transparency and measure incrementality. Warehouse-native identity resolution lets you see and own the matching logic. Audience-level holdout testing measures whether your spend caused results, without relying on person-level matching.

Audio version of this blog (AI-generated)

0:00 / 0:00
Share

Since the early days of cigarette smoke-laden advertising offices (see: Mad Men), marketing and advertising professionals have been attempting to reach the right people at the right time with the right messaging. 

And while that directive hasn’t changed, the technology certainly has. These days, ad and marketing teams leverage their first-party data alongside identity vendors and data onboarding tools to create rich profiles of the customers they’re trying to reach, with the expectation that their advertising will be more effective.

The promise of these identity vendors is relatively straightforward: They take your data, match it alongside their own identity graph (sometimes called an identity spine), and create a more complete profile that helps with ad targeting and measurement. Then, the identity vendor sends that data directly to an ad platform (e.g., Meta). 

These platforms provide high-level visibility of match rate on the ad platform as the metric for success; the higher the percentage of individuals matched to your first-party data list, the better the ad performs. 

But match rate as a KPI can be problematic because it doesn’t answer whether the person being targeted is actually the person matched in the ad platform. Usually, the quality of the match is not accessible because ad platforms rarely report individual-level demographics for targeted individuals, which you could compare to your own customer 360 to assess accuracy. Data onboarding tools don’t reveal the results of the match rate process or the identity spine, sometimes even dropping the known customer data provided before matching. There’s no visibility into how it works or, crucially, whether it works at all.

Match quality concerns

New research published this week on arXiv by Tobi Konitzer, PhD, VP of AI at GrowthLoop, and David M. Rothschild, economist at Microsoft Research, found low match quality in the historical LiveRamp dataset and use case it examined, raising questions about the limitations of evaluating identity resolution services based principally on match rate. The paper also analyzed match rate data for Kochava, a mobile advertising and data broker company, which didn’t perform much better than LiveRamp.

To conduct this research, Tobi and David built a hard-to-come-by golden comparison set, allowing them to assess the quality of the matches from LiveRamp and Kochava.    

In the case of LiveRamp, the research found its delivered matched identities agree with the submitted customers’ basic demographics “barely more often than a coin flipped over a 5x2 table.” While the authors note that the analysis dates to 2019/2020, the reality does not change that advertisers should ask hard questions about match rate and quality. 

The study suggests that, in the dataset and matching process examined by the researchers, a meaningful portion of activated identifiers may not have corresponded to the individuals the advertiser intended to reach. And when an activated audience does not accurately correspond to the intended individuals, some advertising spend will reach people outside the advertiser's intended audience.

Book a GrowthLoop demo

Match rate vs. match quality

Before we dive deeper into the research, it’s important to distinguish between match rate and match quality. 

Match rate is the percentage of contact records that can be linked to identifiers, enabling a demand-side ad platform (DSP) to recognize an individual as a potential ad target. It’s arguably the gold-standard metric for success and differentiation among identity vendors. In almost all cases, a 100% match rate is impossible. LiveRamp claims up to a 52% increase in match rates versus hashed emails.

For example, if you have 100,000 people in your target audience and the match rate is 50%, 50,000 records can be found and used for ads in the DSP. The direct effects of ad exposure can be assessed for these 50,000 people. 

But as this new research shows, match rates are a misleading success metric. Match quality looks at something different: among the identities the data onboarding vendor (such as LiveRamp) returns, how many are actually the person you submitted vs. a stranger who at best may happen to share a household or a device? 

Given industry standards, identity vendors may have an incentive to maximize match rate even if the match quality is adversely affected. 

Let’s build a scenario around this concept. Say you have a customer list of 100,000 records:

  • Vendor A reports a 90% match rate (90,000 customers activated), but the match quality is only 15%. That means you’re reaching 13,500 intended customers and 76,500 other people.

  • Vendor B reports a 40% match rate (40,000 customers activated), but the match quality is 90%. That means you’re reaching 36,000 intended customers and 4,000 other people. 

Using match rate as the metric for success, Vendor B is the “worse” option. But in reality, it’s reaching more of your targeted customers. 

Side‑by‑side charts comparing two ad vendors: one with higher match rate but lower precision, the other with lower match rate but higher effective reachSide‑by‑side charts comparing two ad vendors: one with higher match rate but lower precision, the other with lower match rate but higher effective reach
(From the published paper) Figure 1: A stylized example of high match rate with low match quality. (a) Composition of the activated audience for two hypothetical vendors given 100,000 submitted IDs; the vendor with the higher reported match rate reaches fewer of the intended customers and many more bystanders. (b) Effective reach (share of the submitted list activated and correctly identified) as the product of match rate and quality; iso-reach curves show that very different match rates can deliver identical effective reach.

Where LiveRamp landed on match quality

The researchers dug into the match rate vs. match quality question with one of the industry’s leading data onboarding tools, LiveRamp.

Methodology

The researchers used LiveRamp to match a 2019/2020 national voter file to online identifiers (Mobile Ad IDs, or MAIDs), and then targeted these MAIDs with in-app survey requests (a core LiveRamp offering).

The in-app surveys asked hard demographic questions from the voter file, such as age and gender, to see whether they matched with the voter file. The surveys were conducted with mobile ad IDs that had been successfully targeted with previous poll requests. 

The overall sample of voter file-matched survey responses was 119 for one vendor (LiveRamp) and 1,176 for the other (Kochava) .

Results*

The results warrant scrutiny for any team using LiveRamp for data matching. 

After comparing the age and gender recorded in the voter file with the data collected from in-app surveys, they matched at a rate of 21.8%. Kochava achieved a 12.8% rate.

The researchers use these results to evaluate the extent to which the delivered matches may correspond to the intended individuals. The study points out that if matches were done at random on a 5x2 demographic grid (two genders, five age groups), then the expected match accuracy would be 10% even if accuracy was random. That means LiveRamp and Kochava are matching age and gender successfully only 11.8 percentage points and 2.8 percentage points higher than this random-assignment benchmark, respectively. 

We acknowledge that the study evaluated historical data and particular identity resolution workflows, and its results should not be read as an evaluation of LiveRamp's or Kochava's current technologies, products, or performance. Match quality may vary by product, dataset, matching methodology, use case, and time period. The findings nevertheless illustrate why advertisers may benefit from evaluating match quality alongside reported match rate.

Line chart titled “Implied identity-level precision” showing three declining curves for vendors A and B as demographic agreement rate increases.Line chart titled “Implied identity-level precision” showing three declining curves for vendors A and B as demographic agreement rate increases.
(From the published paper) Figure 5:Implied identity-level precision, both vendors. Identity-level precision π implied by the mixture model in Equation (1) as a function of the demographic agreement rate b of a wrong-person match, for Vendor B (all pairs and individual-level pairs) and Vendor A (audited links). Curves show point estimates. We report no intervals: a sampling interval would hold b fixed. Vertical lines mark the uniform (10%) and empirical (9.4%) chance rates. The curves assume that correct links always agree on age bracket and gender ( e = 0 ); Table 2 relaxes this assumption, which does not change the value of b at which each curve reaches zero.

What this means for marketers and advertisers

The study identifies potential problems far bigger than misplaced advertising spend; it has downstream effects, too. 

If match quality issues like those identified by the study occur in a particular campaign, they could affect downstream activities such as retargeting, suppression, conversion attribution, and clean-room analysis. That means retargeting audiences could be mostly unknown, suppression lists could fail to suppress the intended targets, conversion attribution would credit ads shown to the wrong people, and clean rooms are joining rows to the wrong humans. Of course, the extent of any impact will depend on the particular data, identity resolution methodology, and advertising workflow.

Critically, none of this is likely to show up in your campaign reporting dashboard because reach and match rate look healthy. But because the match quality is hidden inside a black box, you can’t see that the campaign is reaching the wrong audience. 

Ways to evaluate identity resolution

If this research is making you question your current practices, I don’t think you’re alone. But what do you do instead?

One option is to use a data onboarding tool or identity graph with more transparency and access. At the very least, you should have greater visibility into the match logic and quality and you should own that within your data warehouse, not a third-party platform. 

This visibility includes impression data. If you use The Trade Desk, this is called the Raw Event Data Stream (REDS) logs, which are sent from The Trade Desk to your data warehouse and matched at the person or household level. However, this does not ultimately address the problem of match quality vs. match rate inherent in all identity resolution vendors.

GrowthLoop works closely with both TransUnion and Audience Acuity to resolve identity and enrich data. Instead of copying your data out to do that identity resolution and enhancement, it stays in your data warehouse. That can provide greater visibility into the matching process and may reduce the need to move certain customer data outside the customer's existing data environment.**

And if you leverage The Trade Desk for off-site advertising, our deep integration allows you to export data securely and directly to The Trade Desk using the UID2 Framework (without needing LiveRamp in the middle, which may reduce certain data-onboarding costs depending on your configurations and use cases). 

All that translates to more transparency in your warehouse and throughout your campaigns, increasing confidence and visibility that the right people are receiving the right message at the right time. 

An alternative concept of measurement

Another alternative is the incrementality measurement model. This is a framework that randomizes users into control and treatment groups at the audience level, and estimates the causal effect of assigning ad spend against a holdout without requiring person-level exposure matching. 

This model lets you move beyond optimizing an attribution pipeline and answer a more important question: Did the channel or even the overall brand spend drive incremental outcomes? (i.e., what is the incremental effect of showing the ad on this channel vs. not showing the ad?)

GrowthLoop provides incrementality analysis for both Audiences and Journeys. Using AI alongside your customer 360, our platform can create smart splits that keep the control-treatment split dynamically optimal to minimize the cost of the experiment, including dynamic auto-rollout of statistically significant winners. 

Composable data onboarding is the future

Recent industry developments surrounding identity infrastructure have also prompted renewed discussion about how advertisers evaluate and control their identity-resolution workflows. And GrowthLoop believes that composability is the answer.

With a warehouse-native identity resolution, data onboarding solution, or an incrementality measurement model, advertisers have additional ways to evaluate audience activation and campaign performance while maintaining greater visibility into their data workflows. All that translates to more transparency in your warehouse and throughout your campaigns, so you can trust that the right people are receiving the right message at the right time. 

If you want to see how it can help your organization see better ROI from ad spend, reach out to our team for a custom demo.

The measurement hierarchy needs to change

Marketing measurement should happen in a different order.

The first question is does it work? Does this channel, campaign, or intervention create incremental business value? That should be established before we spend time optimizing the details.

The second question is for whom does it work? Which customers, segments, products, markets, or situations actually respond incrementally to the intervention?

Only then should we move to what specifically works? Creative, message, offer, timing, frequency, channel, audience, and sequencing all matter, but they matter much more once we know we are optimizing something that actually changes behavior.

Finally, we can ask where should the next dollar go? At that point, budget allocation becomes much more intelligent because we are comparing the incremental value of different investments rather than comparing the amount of revenue each system has managed to claim.

Notice what is missing from the top of that hierarchy: who gets credit?

That is because credit is fundamentally an accounting question.

Growth is a causal question.

Book a GrowthLoop demo

*The study described in this article examines historical datasets and specific identity resolution workflows. Its findings should not be interpreted as measurements of any vendor's current products or performance, and results may differ across datasets, products, methodologies, and use cases. Unless otherwise indicated, quantitative findings described below are findings of the study, not independent testing conducted by GrowthLoop.

**The study did not evaluate GrowthLoop, TransUnion, Audience Acuity, The Trade Desk, UID2, or the alternative approaches discussed below. We therefore do not suggest that the study establishes that these solutions achieve higher match quality than LiveRamp or Kochava.

Share
More from the GrowthLoop blog

Data + marketing resources

More marketing and data resources

Sign up for the GrowthLoop newsletter

Loading form...
cloud background 4