By: Alva Ree
A conversation about the retention models that make young channels look loyal – and lead companies to scale the wrong acquisition sources.
Interview
In this conversation, Atul Mohan, PhD, examines a deceptively simple modeling error that can reshape marketing budgets before anyone notices what went wrong. With more than 24 years of experience across data science, AI, marketing analytics, and executive decision-making – including leadership roles at American Express, Interpublic Group, Omnicom, Teachers Federal Credit Union, and Brainlabs – Mohan is known for translating complex statistical questions into decisions business leaders can actually use.
What is the statistical error hiding in plain sight?
A statistical error hiding in plain sight is telling marketing teams to scale exactly the wrong channels.
Somewhere in your company there is a model that predicts which customers stay. And there is a good chance it is lying to you in a very specific direction.
Here is how it usually happens. Someone wants to know which acquisition sources produce customers who stick around. They pull a year of data, label every customer as retained or churned, fit a classifier, and rank the sources. The newest channels come out looking terrific.
They are not terrific. They are young.
Why do newer acquisition channels look stronger than they really are?
Think about what that label actually means. A customer you acquired eleven months ago has had eleven months to leave. A customer you acquired last month has had one. If your label is a simple yes or no inside a fixed window, the recent cohort is mostly marked as retained because it has not had time to do anything else yet. Your model dutifully learns that whatever is distinctive about recent acquisition predicts loyalty.
That is a fact about your calendar. It is not a fact about your customers.
Statisticians call this right censoring, and it is the reason survival analysis exists as its own discipline rather than a footnote in a classification textbook. A censored observation is incomplete, not negative. You do not know the customer stayed. You know they had not left by the time you happened to look. Those are different claims, and collapsing them into a single binary label throws away the distinction permanently.
Why does this technical distinction matter commercially?
I want to be blunt about why this matters commercially, because it is easy to file this under academic pedantry and move on to something with a chart.
The bias does not distribute itself evenly. It concentrates precisely on the channels you started buying most recently, which are precisely the channels sitting in front of you in a budget meeting waiting for a decision about whether to scale. This is not random noise that washes out across a large enough sample. It is a systematic thumb on the scale in favor of recency, applied at the exact moment you are most exposed to it.
I have seen teams double down on a source for two quarters on the strength of a retention number that was mostly an artifact of when they started buying it. Nobody was careless. The analysis was competently executed. The code was reviewed. The target variable was just wrong, and nothing in a code review catches a wrong target variable.
What is the correct way to model the problem?
The fix is not complicated, though it does require admitting the first model needs rebuilding. You model time to event rather than whether an event occurred. Cox proportional hazards, accelerated failure time, discrete-time hazard specifications, all of them handle incomplete observations correctly. And all of them will hand you a different ranking of acquisition sources than the classifier did. That difference is the size of the mistake you were making.
There is a second version of this error that I run into almost as often, and it comes from people who already know about censoring. Their solution is to restrict the sample. Only look at cohorts old enough to have a complete observation window, the reasoning goes, and every label becomes honest. Which is true, and it quietly removes from the analysis every channel you have started buying in the last year.
You have traded a biased answer for a clean answer to a question nobody asked. The model is now unimpeachable and silent on the only decision that was actually open. I have more sympathy for this mistake than the first one, because at least it comes from knowing the problem exists, but the practical outcome is the same. The decision gets made on instinct while a very rigorous analysis sits next to it describing a period nobody is asking about.
What should analytics teams do differently from the beginning?
A few things I would push any team to do.
Look hard at whether your target variable has a time dimension you collapsed without noticing. If the real question is when rather than whether, you have a survival problem no matter how the data arrived on your desk or what the ticket said. This is worth asking explicitly at the start of a project, because by the time the model is built, the framing is usually locked in and nobody revisits it.
Plot the curve instead of quoting the rate. A single retention percentage tells you almost nothing. Customers leaving in the first sixty days and customers leaving in the second year are different businesses with different causes and different fixes, and one number flattens both into an average that describes neither. Early churn usually points to onboarding, expectation mismatch, or a product that was sold to the wrong person. Late churn points at competition, price sensitivity, or a value proposition that erodes. Same headline number, completely different investment.
Treat any correlation between your model’s favored segments and acquisition date as a red flag rather than a finding. In my experience, that correlation is almost never real, and the fact that it looks like a signal is exactly what makes it dangerous. A finding that confirms recency bias is emotionally satisfying, because the recent channels are usually the ones somebody fought to add.
And stop treating the censoring indicator as an inconvenience in the pipeline. Knowing which customers have not yet had the opportunity to churn is information. Throwing it away is what created the problem to begin with.
Once time is modeled correctly, what other decisions still matter?
There is a related trap worth naming, because it catches people who have otherwise done everything right. Once you start modeling time to event, you have to decide what counts as the event. In a subscription business, is a downgrade a churn? Is a lapsed payment that gets recovered three weeks later? In lending, is a delinquency an event or a state you can recover from? These definitional choices move results more than the choice of model family does, and they are usually made by whoever wrote the first query rather than by anyone who thought about it.
I would put that decision in front of the business explicitly. Not because analysts cannot make it, but because it encodes a commercial judgment about what the company considers a loss, and that judgment belongs to the people who own the number.
The larger point here is about what you are asking a model to do. A classifier answers a question about the past, inside a window somebody chose arbitrarily. A hazard model answers a question about the future, conditional on what has survived so far. Every acquisition decision you make is about the future. It is worth using the tool that was built for it.
What becomes possible when companies model time properly?
There is one more reason to care, beyond getting the ranking right, and I think it is the one that actually justifies the effort.
Once you model time properly, you can put an expected lifetime value on a customer at the moment they arrive rather than waiting months to find out. That changes what you can optimize. Instead of buying against cost per acquisition, which is the number you can see on the day you spend the money, you can buy against expected margin, which is the number you actually get paid on.
That sounds like a small reframe, and it is not. Cost per acquisition and expected margin per acquisition rank your channels differently, sometimes dramatically. Two sources can produce customers at an identical cost and have retention profiles that differ by a factor of two. Every dollar you allocate to the first number and not the second is a dollar allocated on the basis of something you happen to be able to measure quickly rather than something that matters.
If the methods are mature, why do most companies still optimize the wrong number?
Most companies never make that jump. They know the two numbers are different. Everyone in the room can articulate why. They keep optimizing the wrong one because it is the one their reporting produces on Monday morning, and the right one requires a model somebody has to build, defend, and maintain.
The modeling to fix it is genuinely not the hard part. Survival methods have been well understood for half a century, and the implementations are mature and free. The hard part is organizational. It is being willing to rebuild something that already looked like it was working, and then explaining to a room of people why last quarter’s channel ranking, the one that informed decisions they have already made and defended, needs to be thrown out.
That conversation is uncomfortable. It is also the entire value of doing the work, because a model nobody acts on is an expensive way to produce a document.
About Atul Mohan, PhD
Atul Mohan, PhD, is a data and AI executive whose work spans data science, marketing analytics, measurement strategy, and the translation of complex technical systems into business decisions.
