The Independent is a UK-based digital news publisher that transitioned to an online-only model in 2016, reaching millions of readers globally.

The results

The Independent used Subsets to test whether letting subscribers self-declare their content interests, then personalizing content and newsletter recommendations around them, could improve retention. The team ran the same flow against multiple audiences at once, and two of them ended up telling very different stories.

Audience A:

• Retention rate lift: +3.9 percentage point lift vs control
• +44% more newsletter clicks, +387% more marketing clicks
• +46% more newsletter opens, +217% more marketing opens

Audience B:‍

• Retention rate lift: -6.1 percentage points vs control

The challenge

Like most publishers, a key priority for The Independent is keeping subscribers engaged well before they approach their renewal. The customer marketing team wanted to re-engage subscribers showing early signs of declining engagement, before that decline turned into a renewal risk. Identifying those subscribers was difficult and building several controlled experiments on top of that and tracking engagement and retention over weeks would have required significant engineering resources. 

As a result, the team previously had to rely on broader, one-size-fits-all journeys and CRM-level metrics like opens and clicks, without a reliable way to connect those numbers to actual retention outcomes, or to know whether a tactic that worked for one group of subscribers would work for another.

The solution

Using Subsets, the customer marketing team first identified relevant audiences and then built a 6-part email journey: subscribers with known interests got a weekly newsletter with personalized content recommendations, while those without interest data were first asked to share them. Anyone who still didn't received a weekly round-up of top stories across sections.

See examples of the campaign below. Note: Since this experiment was run, The Independent Premium has transitioned to Independent Membership.

Using Subsets, the customer marketing team could:

• Identify relevant audiences first, rather than targeting the base as a single group
• Run the flow against several audiences at once, rather than testing it on a single group, to see which audience it worked best for
• Track each experiment independently against its own control group, with engagement and retention followed in real time from day one

For one audience, disengaged subscribers a few months into their subscription, the flow worked exactly as expected: newsletter clicks, marketing clicks, and opens all increased, and retention followed with a +3.9 percentage point increase compared to the control group.

For a second audience, the same flow moved retention in the opposite direction: -6.1 percentage points below the control group. Here, Subsets played a direct role in how the team responded:

• Real-time tracking caught the divergence early - well before either cohort reached its full renewal cycle, and before the experiment could scale to a larger share of the base and cause sustained revenue loss
• Subsets flagged that the early results weren't yet statistically significant, so rather than reacting too soon, the team knew to hold off acting on it until the signal was more reliable, especially since the results had looked positive at first before changing significantly

If the experiment had been run across both audiences at once, The Independent would only have seen a blended, ambiguous result, and wouldn't have learned that the flow was working well for one audience and hurting retention in the other. Testing them separately revealed how differently audiences can respond to the exact same flow.

Using Subsets, the customer marketing team could see exactly where personalization was working and where it wasn't, and act on each audience accordingly, using the underlying behavioral metrics from the underperforming cohort as a starting point for what to try next.