In high-stakes situations, you're still going to need a human in the loop for a really long time.
Mahesh PatilDirector, WonomentsAn AI-generated case summary and a recommendation that could affect someone’s freedom demand different levels of scrutiny. Requiring a specialist to check every routine summary can slow operations, while reviewing high-stakes recommendations too quickly can also lead to serious errors in the final decision. Risk-based AI review addresses that tension by matching oversight to the potential harm of a mistake and directing high-stakes recommendations to experts qualified to challenge the evidence.
Mahesh Patil, Director at Wonoments Limited, an AI architecture and advisory firm, has more than 20 years of experience leading technology and engineering initiatives across banking, fintech, and healthcare. His background spans secure cloud platforms, AI-driven systems, and large-scale technology transformations in highly-regulated industries. He now uses AI in coding and agentic workflows but recognizes how easily models can misinterpret instructions or overlook important context.
“In high-stakes situations, you're still going to need a human in the loop for a really long time,” Patil says. He believes organizations can use AI to handle lower-risk tasks while leaving important decisions to qualified experts. These experts can review the evidence and question AI recommendations before making a final decision.
A case summary and a criminal judgment call for different safeguards
The scrutiny an AI output receives should reflect the consequences of acting on it. Language models can produce different responses to the same request, and automated evaluations, including model-based reviews, can't eliminate every error. “Every decision that a clinician makes or a financial expert makes can have far-reaching consequences on people's money and health,” Patil says. The potential harm therefore matters as much as the quality of the model's answer.
Patil points to the Crown Court backlog to illustrate the difference. Nearly 81,000 cases were waiting to be heard, and AI-assisted summaries could help staff handle large volumes of documents. A recommendation that influences a judgment about someone's freedom requires deeper scrutiny. “Some cases you may automatically trust because it's a low-risk use case,” he says. A routine summary may need another level of review if someone uses it to make a consequential decision.
The person approving the answer needs to know the subject
When AI recommendations carry serious consequences, reviewers need enough subject-matter expertise to examine the evidence and challenge the conclusions. “It's not just, oh, you read the output provided by the AI model and you just communicate it,” Patil says. “They need to look at the sources.” That means checking whether the model correctly understood the underlying information and whether its conclusions are supported by the evidence. Patil argues that reviewing AI recommendations requires the same professional expertise as making the decisions themselves. Cardiologists are trained to assess heart conditions, just as financial advisors are trained to evaluate investment options. “You can't put a software engineer and make them build a bridge,” he says.
Understanding how AI works doesn't automatically qualify someone to judge whether its recommendations are correct. Who provides that oversight can vary by industry. Senior domain leaders may be best equipped to review AI-assisted decisions in product-led businesses, whereas healthcare organizations may need doctors and other frontline clinicians who work directly with patients. “The big decisions would still require that specialist expertise,” Patil says. Organizations therefore need reviewers who understand the risks and can recognize when an AI recommendation is wrong.
Faster AI decisions need data that keeps up
As decision-making speeds up, AI systems need information that reflects changing conditions. Patil contrasts traditional three-, six-, and twelve-month forecasts with the shorter timelines enabled by language models. “It enables us to make almost real time decisions even if there may be a human involved,” he says.
He sees growing demand for streaming databases and graph systems that bring in new information and preserve relationships between records. At a Google event, he helped build a small app that extracted details about windows, kitchens, and furniture from property photos and added them to property records. Those details could help match homes to buyers' budgets, locations, and preferences. “How do you enrich your data set based on that to make the right decisions?” he asks. More complete context can make recommendations useful, even when expert approval remains necessary.
Shadow AI leaves employers in the dark
Companies cannot assess AI risks they cannot see. Patil worries that healthcare and financial services organizations may hesitate to approve AI tools, even as employees use personal accounts to access them. “The shadow AI is the biggest risk that organizations are facing today,” he says. Staff may copy business information into external tools and email the results back to their work accounts, leaving employers with little visibility into how that information was handled.
Patil supports giving employees a controlled environment to experiment with AI. This approach lets teams monitor usage, spot risks, and adjust safeguards or investment choices. Giving employees approved ways to experiment with AI also helps leaders identify which tasks need expert review. Patil believes organizations should embrace AI while setting clear rules for its use. “AI is here. It's best we start to figure out how to work with it.”
