AI is outpacing guardrails. Businesses must build their own

fastcompany.com

Dario Amodei, the CEO of Anthropic, recently joined the chorus of Silicon Valley Cassandras calling on AI companies to deliberately slow the pace of product development. In a widely discussed letter, he predicted a nightmare scenario in which a swarm of AI agents takes over the internet and causes hundreds of billions of dollars in damage—something he thinks could happen within the next six to 12 months. But we don’t have to wait a year to see the damage artificial intelligence can cause if ungoverned. Most near-term harm will come not from rogue agents but from ordinary systems, bought from ordinary vendors, making ordinary decisions about people. For example, in the banking industry, AI is already helping decide who gets a loan and at what price, whose claim is flagged, and whose account is closed. According to Deloitte, twice as many bank employees use AI weekly as did a year ago and the number of publicly reported AI incidents and hazards in the sector was eight times higher in 2025 than in 2022. That might not feel like an existential problem, but it has real consequences for people’s finances and well-being—and the same problem exists across health care, transportation, and other critical sectors. The labs may be able to pace what they build, but not what the rest of us deploy. Nor are regulators moving fast enough to protect your business, your customers, or your shareholders’ interests. Every business must decide for itself what it will and will not let a machine decide. A framework to fill the regulatory vacuum The potential consequences of ever-increasing AI uptake plus lagging governance are clear. An agentic system connected to internal infrastructure can open the door to cyberattack. An employee untrained in AI can inadvertently perpetuate hallucinations or misinformation. An underwriting algorithm based on biased historical data can quickly deny qualified borrowers. Real-world impacts like these require more than a page of AI use guidelines that gets posted on an intranet and never referenced again. Over the last year, I’ve worked with 14 banks across the Global Alliance for Banking on Values to build and test a comprehensive framework to guide AI governance in this regulatory vacuum. While the need is urgent in banking, this framework applies equally to other critical sectors, including health care, insurance, retail, and telecommunications, where AI-enabled decision making intersects with human lives. Business leaders should start by acknowledging that AI needs guidance in a way other technologies do not. AI decides rather than implements. It reasons, but that reasoning is a black box. For that reason, AI governance can’t just be another risk register where your board reviews a list of things that can go wrong and how likely that is to happen. AI governance requires business leaders to contend with the good a business has failed to do and the customers who were never reached because the model failed to see them.  In practical terms, there are three immediate steps that business leaders can take across sectors to start building that moral architecture, all revolving around amping up human oversight at every step: 1. Name an owner for every AI system Start with an inventory of where AI is doing the work. You cannot govern what you have not listed, and institutions often find more AI embedded in their work than expected. Plenty of AI enters a company without ever passing through model review, arriving instead through vendor software or tools employees adopt on their own. So the first step is a complete inventory of every AI system operating in the institution, no matter how it got there. Each system then needs a named owner who has real authority. That person must have the standing to enact changes, responsibility for periodically revalidating the system, and, if necessary, the ability to take the system offline. 2. Keep human judgment in the loop Hospitals are increasingly using AI to facilitate diagnoses, but a qualified medical expert is always the one to make the final clinical decision and explain it to the patient. The same must go for banking or any other business. Employees using AI systems need training to evaluate what those systems produce and feel empowered to override outputs. They must also be able to explain the final decision to both customers and regulators in plain language, without an engineer translating. Track how often employees override AI systems. If a system has run for a year and no one has ever disagreed with it, human judgment is lacking. 3. Include vendors in your governance strategy Most institutions buy AI rather than build it, so governance must travel upstream into every contract. Fully 91% of senior executives in a recent survey said that they don’t fully understand their organization’s dependencies across AI vendors, models, and infrastructure. The concentration on a few foundation models is such that Moody’s recently raised the alarm that if just one of those providers has an outage or raises prices, the effects could ripple across the entire banking sector—a risk that applies to other sectors as well. Make sure to embed vendor accountability into your AI governance frameworks. That means including things like audit rights, bias-testing support, incident notification, and the right to terminate in all vendor contracts. It’s not all doom and gloom Pessimistic predictions about AI make headlines, but the optimistic vision is the strongest argument for taking action. AI has the potential to reduce customer friction, make space for employees to take on more meaningful work, and expand economic opportunity. For example, data across more than 180 banks and credit unions using Zest AI’s machine learning model—which uses alternatives to traditional credit scores to assess risk—shows 49%, 41%, and 40% increases in loan approval rates for Latino, Black, and women borrowers, respectively. Far from being a tax on innovation, proper AI governance can help broaden market share and reputational value. None of that happens automatically. When regulators do eventually catch up, they’ll either be codifying what responsible companies have already built or cleaning up after the ones that waited to be told. Businesses get to decide right now which of those they’ll be, and every one of us has a stake in the answer.