The Case For Open Source Grows As Regulated Industries Extend AI Governance Into Workflows

Data governance in regulated industries runs on established practice. Encryption and access logging are standard, and auditors know what to ask for. No equivalent standard covers what happens after a record leaves the database. It passes between software systems, a model reasons over it, and a person acts on the answer. Adding checks at those points means editing the software that runs them. An organization can only edit software it owns, which is why open source infrastructure is becoming an AI governance requirement.

Vladimir Filkov is a Professor of Computer Science at the University of California, Davis, where he studies how open source projects govern themselves and what keeps them viable over time. He directs the DECAL Lab and AI for Health Lab, and is the Director of the AI for Health Initiative at UC Davis. He also helped establish the University of California network of Open Source Program Offices, encompassing six UC campuses. His current research designs governance systems for workflows that software agents and people run together. He also applies AI in clinical settings, where a missing control has consequences a patient can feel.

"Checks have to be not just on the data, but on the passage of data between systems, on the reasoning itself, and on the way AI interacts with people," says Filkov. Every database has an owner and an audit trail. The three steps he describes fall between systems, where responsibility is rarely assigned to anyone in particular.

Where liability lands

When an AI system contributes to a decision that harms someone, the liability belongs to the organization that deployed it and to the professional who acted on its output. Regulators and courts have not created a category for the software itself. Responsibility stays where it was before the model arrived, which is why accountability keeps landing on named individuals rather than on systems. "AI cannot be liable. It has to be the person that is using it, the person who benefits from it," Filkov says.

Those same people are expected to catch the system's mistakes, and the tools rarely give them extra time to do it. A doctor using an ambient documentation tool still sees the same number of patients, so the note review has to fit into a day that was already full. A loan officer reviewing an AI-drafted credit memo has the same number of files to close. Filkov expects those reviews to get faster and less careful as the tools become routine, a pattern already visible in clinical validation work elsewhere in health AI.

Filkov's answer runs underneath the human review. "Downstream of the AI there have to be checks that are more deterministic, traditional algorithms that catch mistakes with dosage, medicines, and allergens," he warns. These checks contain no AI. They are ordinary rules, written by people and applied the same way every time. A credit system compares a proposed exposure against a fixed limit. A clinical system compares a prescribed dose against a patient's weight and allergy list. Neither asks the model anything, so a hallucinated dose fails the same test a mistyped one does. Building that layer means adding a step to the pipeline the record travels through. Policy documents cannot do that work, which is part of why enterprises are moving toward agent control planes that enforce rules in the runtime itself.

The cost of oversight

Filkov's lab asked an AI to write three governance documents. One covered AI systems working alone, one covered people working alone, and one covered both working the same job together. The document for the third case came back longest. The model explained why in its own draft. It works faster than the people reviewing it, so full review turns humans into the slowest part of the process.

An organization has two choices. Reviewing everything keeps the work moving at the speed of the reviewer. Sampling requires deciding in advance what gets skipped, and being able to defend that decision to a regulator. "If you are going to check on every patient after the AI is done, you are not benefiting. You may as well see every patient," he says.

Filkov's answer is to sort the work before a person ever sees it, so that only the cases needing judgment reach someone qualified to apply it. "The AI performs some of the duties of the doctor for easy cases, and only the very few hard cases are sent to a specialist," he adds. The sorting rule is now the part that needs governing. It decides which cases get a person's attention, and an organization that cannot explain how it draws that line cannot defend any decision that follows from it, a problem insurers have run into while aligning human judgment with automated triage.

Open source at scale

Open source projects have no boss. The people writing the code work for different employers in different countries, and nobody can order any of them to do anything. Governance has to be explicit for the project to function at all. Filkov's lab studies which projects hold up under that arrangement, and the ones with written rules outlast the people who started them.

Estonia runs its government on an open source data exchange layer called X-Road, which moves records between ministries, hospitals, banks, and the tax authority. The country can read that code and change it. Finland runs the same software, and the two governments jointly fund its development through a shared nonprofit. The reason is continuity. A national health record system built on software a country controls keeps running if the company behind it raises its prices or leaves the market. "Many countries are taking open source as a sort of insurance, a sort of parallel line for sustainability for their own country's government," says Filkov.

The same reasoning applies to a single organization. A change to proprietary software starts with a request to the vendor and lands whenever the vendor schedules it. A change to open software can go straight to the organization's own engineers and be finished by the date the regulator set. That difference decides whether the checks Filkov describes can be added at all, and whether a transfer between two systems can be logged. Sovereignty over data comes down to who is allowed to edit the code.

Hiring for open source

Owning the software only helps if someone is paid to work on it. That job belongs to an open source program office, or OSPO, a small team sitting inside the organization. The team decides which outside projects the company is willing to build on and tracks the license terms attached to each one, since those terms vary from project to project. It reviews that code for security problems. When the outside developers release a new version, it confirms that the company's own modifications still work. Google runs one. University of California researchers and librarians developed a six-campus OSPO network funded by a grant from the Alfred P. Sloan Foundation, with Filkov co-leading the office at Davis. "What an OSPO allows is the organization to understand how to use open source in its business," he adds.

Whether that work has been done shows up the moment something has to change. A regulator issues new documentation rules. A check written for one data field has to start reading another. An audit finds that records moving between two systems left no log behind. Each of those is a code change. The organization's ability to make it was decided years earlier, when it chose its systems and set the headcount to maintain them. "It is amazing how much you can achieve if your whole healthcare stack is open source and you can go in and tweak it," says Filkov.