In The News
-Sathish Raman
Subtitle/Image caption – A machine learning leader argues that the industry’s biggest production problem is not the model. It is what happens to the model after it is built.
Most enterprise ML systems decay after deployment, a problem Smeet H Patel calls the closed-loop gap. This article reveals why models stop improving and introduces RetentionOS, a framework for continuously self-improving ML systems. Learn how to implement causal design and prevent your AI from silently degrading, ensuring long-term effectiveness and ROI.
Most enterprise machine learning systems perform best on the day they ship. From there, they get worse. The model trains on the world as it was, goes live, scores customers, drives decisions, and the world it was built for keeps moving without it. Smeet H Patel calls this the closed-loop gap, and he believes it is the structural reason so much production ML quietly decays while the teams that built it assume it is still working.
The gap is simple to state and easy to miss. A retention model predicts who is likely to leave. It triggers an intervention. An outcome occurs. That outcome lands in a data warehouse and stays there, permanently disconnected from the model that caused it. “The model learns only from what was true before it was built,” Patel says. “It does not learn from what it caused.” Open-loop systems cannot improve from their own decisions, so they drift. The fix, in Patel’s account, is not a better algorithm. It is architecture that feeds outcomes back into the system that produced them.
That conviction is the foundation of RetentionOS, a framework Patel formalized in 2025 out of years of running prescriptive retention systems at a large enterprise telecommunications provider, where subscriber retention operates across tens of millions of accounts. The framework integrates five stages that most organizations build and run as separate workstreams: churn prediction, uplift-based treatment assignment, channel optimization, causal outcome attribution, and feedback-loop integration. To the best of Patel’s knowledge, it is the first peer-reviewed contribution to formalize all five as a single, continuously self-improving loop rather than a row of disconnected components.
Closing that loop forces a harder question than the one most churn systems are built to answer. The usual question is who is likely to leave. Patel argues that is the wrong target. “High churn probability and high treatment responsiveness are independent properties,” he says. A subscriber who has already decided to go is a poor investment no matter how confident the model is about the risk. A subscriber with moderate predicted churn who happens to be highly sensitive to a particular offer is where retention spend actually pays off. Optimizing for prediction accuracy instead of treatment lift, in his view, systematically misallocates the budget.
The payoff for getting this right is documented in the field. Peer-reviewed work on well-designed uplift modeling reports treatment-effect improvements of roughly 15 to 30 percent over standard churn-score targeting, and production implementations of the architecture have landed inside that range. At the scale of a major subscriber-based service company, each retained account carries recurring revenue across a bundle of services, so a marginal reduction in voluntary churn within a treated population compounds into material annual revenue. Patel is deliberate about not publishing internal figures, anchoring the numbers instead to the open literature rather than proprietary results.
Building the loop correctly meant solving problems that sit outside the standard machine learning pipeline. The first is attribution latency. A customer who receives an intervention in week one may not confirm whether they stayed or left until week six or eight. Close the feedback loop before that window shuts and the model trains on an incomplete, misleading signal. Patel’s approach buffers the attribution queue by outcome class and holds causal signal out of training until the observation window closes for each treatment cohort. The mechanism is easy to describe and difficult to implement at production scale, and skipping it is how a closed-loop system silently degrades into a worse-performing open-loop one.
The second problem is subtler and rarely corrected in production: the winner’s curse in uplift modeling. When a model picks high responders and those customers get treated, the next training cycle observes a population biased by the previous model’s own choices. Left alone, the model collapses toward a narrow segment and loses its read on the broader treatable base. Patel built a correction that watches the treatment-assignment distribution across cycles and reintroduces diversity before the collapse sets in. Most systems skip it, he notes, because it requires instrumenting the assignment distribution rather than just shipping predictions.
The hardest problem turned out to be human. Operational teams whose metrics and instincts were calibrated around a churn score had to start acting on a treatment recommendation, a channel assignment, and a causal confidence interval instead. “A closed-loop machine learning system deployed into an open-loop operational culture will consistently underperform its technical capability,” Patel says. Every downstream workflow had to be redesigned alongside the model.
His advice to practitioners runs against the instinct to add complexity. Before building a more sophisticated churn model, he says, instrument the outcome attribution first, because most organizations cannot say whether last quarter’s interventions actually caused the retention they observed. “Close the loop first. Then improve the model.” He sees the wider field moving the same direction over the next three to five years, from predictive design toward causal design, “not because of better algorithms, but because the
field is asking better questions.” The organizations that build closed-loop causal feedback now, he argues, will run systems that compound in effectiveness while everyone else pays rising retraining costs just to stand still.
This article has been prepared by our editorial team based on the information provided. The final published version may be subject to editorial changes at the discretion of the journalist and publication. This is a draft for review purposes.
