As AI Applications Multiply, Kubernetes Takes on a Larger Governance Role

Kubernetes spins up a workload in seconds. The databases that workload queries are shared, and they hold records that outlast the application asking for them. AI has multiplied the number of applications requesting those records, and a growing share of them come from employees who don't write code. The orchestration layer creates workloads faster than the data layer can serve them, which leaves the platform to decide what each one is allowed to query. Rate limits, egress rules and read-only access now get applied where the workload is scheduled.

Surya Jayanti is Director of Software Engineering at Walmart Global Tech, where his work focuses on cloud and AI platforms and his experience includes operating Kubernetes platforms across complex enterprise environments. His work covers the platform layer that hosts application workloads with a focus on platform engineering, application infrastructure and data-access patterns. Jayanti's career spans more than 20 years across Walmart, Aetna, Infosys and BP, including engineering organizations he has led in the United States and India. His recent work has also explored AI-assisted developer experiences and platform operations.

"The biggest opportunity is in the rest of the workloads that are newly getting created. The rate at which they are getting created is 10x, rather than 10%," says Jayanti. The new workloads are not the ones consuming GPUs or requiring hyperscale capacity. Most serve a small group of people and exist to answer a single question.

Apps from outside engineering

The workloads that do consume GPUs get most of the attention in enterprise AI planning, and Jayanti's teams run those as well. His team pairs private cloud with public, keeping work on private capacity wherever it fits. Hardware decides which workloads have to leave private capacity. "It cannot scale you to the world of GPUs," Jayanti explains. "You cannot lay your hands on latest and greatest set of GPUs. Only the hyperscalers can give you that, and also the scale at which you want to operate."

In Jayanti's experience, GPU-intensive workloads can represent a relatively small portion of the broader application estate. Training a new model or absorbing ten thousand orders a second describes a narrow class of system. The applications multiplying fastest need ordinary compute and a database that is already running, and most come from people whose job is not engineering. "So far they still need not realize what Python package is running, what the matplotlib library versions I am running, or how the integration is going to work with BigQuery from this Python app," he notes.

A platform that hides its own internals changes who counts as a developer inside large enterprises. Jayanti expects the number of applications built by people outside engineering to keep climbing, which could leave traditional engineering teams focused on a smaller set of specialized applications. "More than 80% of the apps you can run with English," he says. "The other 20% are mission critical business logic, your regulatory compliance, and where you need to scale and really know what you are modeling after."

Short lifespans and idle compute

Applications built outside engineering accumulate quickly, and the estate would keep growing if each of them stayed. Jayanti tracks net additions separately from the rate of creation, and the two numbers diverge sharply. "More than 80% of these apps are short-lived, so they might disappear within three months," Jayanti points out. An organization that provisions capacity for that population without releasing it afterward keeps paying for compute that has stopped doing anything.

Organizations leave the capacity running because they expect applications to last. Enterprise software once did, with ownership passing between engineers over the years and maintenance treated as a permanent line item. The people building applications inside enterprises now are often the ones who needed the answer in the first place. "The same person who created that app last year might retire it this year," Jayanti explains.

Jayanti sees the short lifespan as the useful part. Someone who suspects a particular product will perform better with a particular group of customers can pull historical records, build an application to test the idea, and put the result in front of a leader who makes a decision. Spreadsheets did that work before, and they limited how much history one person could examine. "Your value will come out of that decision," he adds. "Your value will not come out of this short-lived app or the data that it is accessing."

Read pressure on shared databases

A twentyfold increase in applications reaching the same records would ordinarily imply a comparable increase in what is stored, that relationship does not necessarily hold in enterprise environments. The applications arriving now put a different kind of demand on the estate than the ones it was sized for. "Fortunately the trend that we are seeing is they are not creating new data as much as we expected, but they are over enthusiastic in consuming the existing data," he says.

Read-only consumption produces a distinct load profile. Jayanti has watched the same behavior across applications and enterprise environments he has worked with, regardless of what each was built to answer. "They are not creating new access patterns or new ways of consuming or modifying data," Jayanti observes. "It is more like reading the data, showcasing it a little bit differently, providing another point of view."

Reading is still work for the database. The same tables now serve many short-lived workloads at once, and every new application adds callers to capacity that was already allocated. Planning for that has little in common with planning for storage growth. "All these apps are compute-hungry and data access read-only-hungry," he notes. "They are not creating huge net new data."

Limits at the platform layer

A read-only workload can still exhaust a connection pool. Protecting the database when many of them run at once comes down to what the platform permits before a query is ever issued, and leaving each builder to set their own limits wouldn't work at that volume. Access control tends to settle at the platform layer, where it can be applied once and inherited by everything running on top. Jayanti points to a familiar set of guardrails in enterprise environments, including egress rules that govern what can leave, read-only access to large production databases, and a capped share of allocation for any single workload.

Two enforcement points do that work, and both sit outside the application code. Controls placed there survive the applications they govern, an advantage when most of those applications are gone within a quarter. Jayanti describes two common approaches. One caps requests per second at the AD group level, so every application tied to that group inherits the same ceiling. The other applies the limit at the sidecar in Kubernetes, where the constraint travels with the workload.

Most of these applications serve ten people or fewer, so the limits rarely affect whoever built one. Responsibility for the data estate stays with the platform team, where it was before the application count started climbing. "As a platform provider, you need to provide those default governance and controls for the sidecar, or at application onboarding time," Jayanti concludes. "The people using this are part of the AD group, and they cannot consume more than 10 calls per second or 100 calls per second."

The views and opinions expressed are those of Surya Jayanti and do not represent the official policy or position of any organization.