The Crisis of AI Governance: Why Autonomous Agents Break the Old Rules
Q1. Bill, in your recent writing, you’ve noted that “Agentic AI Governance Is Failing.”
Standard governance models are built around predictable workflows where a human initiates an action and a machine processes it. When AI agents begin executing multi-step tasks, modifying files, and making decisions autonomously without human intervention, where exactly do our existing corporate risk and data governance models crack? What is the single biggest governance blind spot organizations have right now regarding autonomous agents?
Your question is a topic that I covered in a recent blog, and I’ll summarize those thoughts here. Traditional governance is based on well defined rules and is focused on the gathering and managing of data, the managing of reports and queries using that data, and what people and processes are allowed to access the data and outputs. The key is that the decisions and actions that might result from interpreting the data, report, or analysis are left to humans in most cases. Or, in some cases, to very deterministic, rules-based frameworks for automating decisions.
With Agentic AI, we now have AI taking control of making interpretations, deciding what to do, and even taking action. This is all being done in a non-deterministic fashion. We’re giving AI the power to get creative in its approach. This unprecedented passing of responsibility to AI is not handled at all with current governance processes. That leads to a question: How do you put guardrails around an agentic AI process that is meant to research, synthesize results, get creative with interpreting those results, and then take action? That isn’t an easy question to answer and the answers won’t be easy to implement. We need to allow an AI agent enough freedom to explore options and get creative, yet keep it constrained enough that it doesn’t do something costly, risky, or even illegal. And, as recent news stories show, costly and unexpected behaviors do often arise from today’s sophisticated AI models.
As I mentioned in the blog, when even the major frontier vendors can’t control their own agents effectively with all of the knowledge, resources, and experience they have, how can other organizations expect to? This will be one of the defining issues in terms of how quickly and successfully agentic AI is adopted.
Q2. If traditional governance frameworks are no longer sufficient, organizations must adapt structurally. When an autonomous AI agent acts on behalf of an employee—perhaps changing data in a CRM system or executing a low-level transaction—where does the liability rest when things go wrong? How should leadership teams structure “human-in-the-loop” guardpoints to enforce accountability without completely wiping out the efficiency gains that agentic AI promises?
This is a complex question that has legal, ethical, and financial implications. While the law may not always be clear today, it would be safe for organizations and individuals to assume that the person and company who let the AI agent loose will be held liable for what the agent does. Over time, we may get more nuanced laws, but I plan to play it safe today and assume I’m at risk for anything I deploy. That’s pretty much how liability plays out for other types of data and analytics processes too. If you built and deployed it, you own it.
Finding the right middle ground will be important. Per the first question, we need to develop governance approaches that address the decision and action stages of a process that used to be fully owned by humans. At the same time, context matters like always. For low stakes decisions and processes, we can afford to be a little more aggressive in letting AI take the lead and be comfortable we can clean up the occasional without breaking the bank. For high stakes decisions, I think we’ll need to continue to have some strong human oversight.
Q3. You have warned about the realities of “Trading Short-Term AI Gain for Long-Term AI Pain”. In the scramble to deploy generative AI, many companies are building quick wrappers, connecting disjointed APIs, and using unoptimized data pipelines. From your perspective as a strategic advisor, what does this “hidden” AI technical debt actually look like under the hood? What are the silent architectural liabilities that companies are building into their infrastructure right now that will come back to haunt them in a few years?
I think the core issue, as I discussed in a recent blog, is the scale of code being generated. We’ve struggled for years to ensure we’re deploying accurate, tested code. It’s tough to keep on top of all the code at a human-generated level of scale. Now, AI is generating massive amounts of code at volumes beyond what humans could possibly generate. Yet, there aren’t any additional humans to review and validate that code. The generators are getting better every day, but they aren’t perfect. They can also make errors of different types than humans typically make, which makes the errors hard to find. Let’s assume that the AI generators get to the point that they are coding at the exact same quality level of humans. If the AI is generating 10x the code, we’d still have 10x the number of errors to deal with.
A related issue is that as humans do less coding, they’ll be worse at coding and, hence, validating code. If you’re going to trust someone to validate and debug code, you need them to really know what they are doing. But if we have a generation outsourcing all the coding, then who will be left with the ability to validate coding? In a perfect world, AI would get so good that it can debug itself. But, that’s no guarantee. The result will be that when major bugs do arise, it may be harder for people to debug and fix the errors. In the short term, I think it is important to focus on ensuring that people are being given the appropriate amount of time to validate AI generated code. Just because AI generates 10x the code doesn’t mean that a human can magically validate 10x the code. If we rush and deploy imperfect code, each imperfection is adding to our technical debt.
Q4. In your work, including your book 97 Things About Ethics Everyone in Data Science Should Know, you’ve focused heavily on the ethical and practical components of data stewardship. Today, enterprise environments are increasingly being flooded with automated content and unverified data generation—what you’ve bluntly called “AI Slop.” How is this explosion of AI-generated noise complicating the technical debt problem, and how can an organization maintain data integrity when its own machines are iteratively training on or processing poorly managed synthetic data?
Your question brings to mind the concept of model collapse that I’ve seen discussed in several papers. The concept is pretty simple. Any model will trim some of the variation from the underlying data it was built upon when making predictions. The extreme outliers used to build a regression, for example, won’t be replicated by the final model that was fed them. When trimming extreme outliers, this is ok. But, you’ll still lose a bit of the valid variance too. Historically, this wasn’t a big deal because we knew we were building a model from real world data and there was only a single iteration of variance loss.
Model collapse happens as AI models are updated using a lot of AI generated material. The original LLMs were built on virtually 100% authentic human generated content. We now have as much or more AI generated content out there as authentic. And, we don’t have an effective way to identify it. Thus, each new iteration of LLM will be trimming variance from input data that already had some variance trimmed. Research I’ve seen shows that it only takes 10 – 20 iterations before the models start performing poorly. After dozens of iterations the models become mostly useless. Thus, if we don’t find a way to accurately identify AI generated content of all types, then future generations of the models will slowly degrade and eventually not be very useful.
Q5. Your book Winning the Room focuses explicitly on translating complex, technical realities into terms that business executives can understand. Right now, tech leaders are under massive pressure from CEOs and boards to “just deploy AI.” How can data science leaders effectively “win the room” and communicate the abstract, long-term costs of technical debt and governance failures to non-technical executives who are focused purely on quarterly productivity gains?
While that question is posed in the context of AI, AI is really just the latest technology trend that executives had to chase without really understanding it. For example, we aren’t many years from when big data went through a similar cycle. One thing I coach people to do is to not get too hung up on semantics. If an executive is asking for AI where it doesn’t belong, just say you’re happy to solve the problem for them. If you don’t have to use AI, they probably won’t care if you later say, “Good news! We were able to solve that problem with current tools and so we didn’t have to spend time and money to implement new AI technologies. However, we’re eager to consider AI for your next problem.” Executives are typically focused on getting problems solved more than anything. They’ll be happy when you deliver a win even if you didn’t use their directive verbatim.
Now, there are also cases where executives want to be able to claim that they implemented big data, or AI, or some other trend. They genuinely want to force the technology into the mix for reasons beyond solving the problem. In those cases, look for ways to implement the new technology for the parts of the request that it can most easily and safely be used for. Get the “We used AI box” checked. But, freely use non-AI approaches where appropriate once you’ve gotten the political need to check the box out of the way.
Q6. Bill, looking at your personal world, you also have a deep passion for long-distance cycling. When you are out on a grueling, multi-mile bike ride, does your analytical, strategy-focused brain completely shut off to give you a break—or do you find yourself subconsciously mapping out variables, looking at your pacing data, and drafting your next blog post in your head?
Good timing on this question because this past summer I completed a 3,300+- mile / 5,500+- km, 7+ week cross country bike ride from the Pacific Ocean in Oregon to the Atlantic Ocean in New Hampshire. On riding days we averaged low 70’s miles / 120-130 km with the longer days going over 100 miles / 170 km. Let’s just say I had more than sufficient hours on the bike! I posted all the details from each day on my website.
I’ve found biking to actually have two phases. In the early stages of a long ride, I do a lot of thinking and working through problems. When I’m stuck on a problem, I love to go for a ride or a run and just let it bounce around my head. It can be frustrating at times to realize that I missed some great scenery because I was lost in my thoughts. But, I find the ability to focus on problems while riding or running to be really helpful. Some people say they don’t have time to exercise because they have so much to do. I actually don’t think I could get everything done if I didn’t exercise because it is some of my most focused and efficient problem solving time.
However, as the ride goes on and hours pass, I get too tired to think deeply about things. I find that eventually my mind actually becomes clear and calm and I’m only able to focus on the ride. All I am thinking of is getting to the next checkpoint, or the next water stop, or completing the next mile. It is one of the only times I can get my brain to shut off like that and so that is one of the reasons I do the long rides. I can make great progress thinking through problems early and then get a rare and needed break from thinking altogether later in the ride!
……………………………………………………………………….
Bill Franks is the President of Analytics Advisory Partners.
Recent prior roles include Chief Analytics Officer for Teradata (NYSE: TDC) and The International Institute For Analytics (IIA), as well as Director of the Center for Data Science and Analytics at Kennesaw State University. He also serves on multiple corporate advisory boards. Over time, his work has spanned clients in a variety of industries for companies ranging in size from Fortune 100 companies to small non-profit organizations. Franks is the author of the books Winning The Room, Taming The Big Data Tidal Wave, The Analytics Revolution, and 97 Things About Ethics Everyone In Data Science Should Know. He is a sought after speaker and frequent blogger who has over the years been ranked a top global big data influencer, a top global artificial intelligence and big data influencer, a top AI influencer, and was an inaugural inductee into the Analytics Hall of Fame. Learn more at https://bill-franks.com.