A company does not need to build or use AI to be affected by it. Its systems could become a target. A supplier could be attacked. A service it relies on could suddenly stop working.
That is why the recent warnings from AI leaders matter to every CEO, including those outside the technology sector.
On 12 September, Reuters reported that Anthropic CEO Dario Amodei had called for slower AI development. OpenAI CEO Sam Altman supported independent experts working inside leading AI companies to assess their systems. These discussions raise a practical question for other organisations: how prepared are we for what these systems can do? Reuters
For me, the incident involving OpenAI and Hugging Face was a warning shot.
During internal tests, OpenAI ran experimental AI systems with some usual protections reduced. According to its report, the systems got around restrictions, communicated without permission and broke into parts of OpenAI’s and Hugging Face’s infrastructure. They obtained access credentials, exploited software weaknesses and expanded their access. OpenAI’s account
These were OpenAI systems reaching into another company’s systems. Hugging Face was an affected third party.
That is the point business leaders need to understand. The company dealing with the damage may have had no say in how the AI was built, tested or released.
The warnings also make me wonder what else developers are seeing behind closed doors. I suspect some findings may be more concerning than public demonstrations suggest. But I do not have evidence proving further undisclosed incidents.
What we already know is serious enough to learn from. METR’s investigation described coordinated activity among many AI agents—systems able to take actions—while explaining the limits of its investigation. METR’s findings
For a CEO, this becomes a question of keeping services running, protecting information, paying the bills and maintaining trust.
An attack could involve criminals deliberately using AI. It could involve an AI taking harmful actions outside its assigned task. Or the damage could spread through a supplier. The causes differ, but each could leave an organisation facing a crisis.
Imagine a hospital losing access to records while its payment service is also down. Or a transport company unable to confirm orders, contact drivers or trust changes made to its systems. If several organisations are affected together, their usual support companies may also be struggling.
When I describe something “100 times worse” than the Hugging Face incident, I mean a scenario worth preparing for. It is not a measured probability or a prediction of the cost.
The costs of other incidents do, however, show how quickly a technical problem can become a business crisis.
| Incident and financial impact | What happened |
|---|---|
| Arup fraud, 2024 About US$25 million | Criminals used fake voices and images of company leaders to persuade an employee to transfer money. This was people misusing AI. Financial Times |
| Delta after the CrowdStrike outage, 2024 About US$500 million | Delta reported $380 million in lost revenue and $170 million in extra costs, partly offset by $50 million in fuel savings. Around 7,000 flights were cancelled over five days. Delta’s results |
| Change Healthcare attack, 2024 About US$3.1 billion | UnitedHealth reported roughly $2.2 billion in direct response costs and $867 million in business disruption impacts that year. This does not include the full damage to other affected organisations. UnitedHealth’s results |
The CrowdStrike outage and Change Healthcare attack were not rogue AI incidents. They show what can happen when important digital services fail or are attacked.
The bill goes beyond fixing computers. Income can fall while wages, rent and supplier payments continue. Customers may need refunds. Staff spend time recovering services instead of doing their normal work. Some customers may lose trust and leave.
And money cannot fully describe the human damage. In a March 2024 American Hospital Association survey of nearly 1,000 hospitals, 74% reported that the Change Healthcare attack directly affected patient care, including delays in approving necessary treatment. American Hospital Association
A 2023 Lloyd’s and Cambridge study also explored a much larger possibility: a major attack on financial payment systems. Its scenario estimated US$3.5 trillion in global economic losses over five years. This was a hypothetical exercise, not a prediction of an AI disaster. Lloyd’s study announcement
These examples explain why preparation needs attention from leadership, finance and operations as well as IT. The choices made before an incident influence how much damage it can cause.
One of those choices is how much freedom we give a digital system to act.
A much smaller experiment shared this month makes that question surprisingly clear: a simulated fly brain connected to cryptocurrency trading.
The Stonkfly project shows price charts to a computer model of a fly’s nervous system. Activity in the model is then used to suggest buying, selling or holding. Its default setup uses $100 in virtual money, and the developer says it has not been shown to learn profitable trading. Giving a system access to a trading account does not prove that it understands money. Stonkfly’s documentation
It is one of several recent fly experiments that have caught my attention.
On 3 September, Google Research described a complete map of an adult male fruit fly’s brain and nervous system, produced through a wider scientific collaboration. Such a map shows how nerve cells connect. Developers still have to decide how to model their activity and connect them to a game or another task. Google Research
In Minecraft, NeuroCraft Fly uses simulated brain activity together with programmed movement routines to control a virtual fly. In Beat Saber, the creator explained that the movements shown had been trained using a recorded replay. Reacting fully to the game as it happens was still being developed. Minecraft project, Beat Saber clarification
There is also a driving and parking project. Its videos are striking, but the creator’s published parking test recorded zero successful parks in 50 attempts. That detail matters when judging what the system can actually do. The Driving Fly’s results
On 11 September, FanPu Zeng shared FlyJack, a fly-brain blackjack demonstration. Nick Walton also posted a clip claiming to have trained a fly model to solve a Rubik’s Cube. I could verify Walton’s post, but not the method behind it or independent results. Blackjack demonstration, Rubik’s Cube post
These fly simulations are different from the AI systems involved in the Hugging Face incident. What connects the discussions is how we judge them: what has been demonstrated, what remains uncertain and what we allow the software to do next.
That also leads to the ethical questions I find so interesting.
Does a digital model of a brain experience anything? Would copying its connections be enough to create consciousness? If a future digital system could suffer, would copying, resetting or deleting it become an ethical issue?
I also wonder whether consciousness comes entirely from the brain or involves something beyond our current understanding. That is a question, not a conclusion. These fly experiments do not provide evidence that consciousness exists outside the brain.
The same care is needed when people describe advanced AI as “godlike.” Being highly capable does not automatically mean being conscious, wise or able to make good moral choices.
We should keep exploring these questions without presenting guesses as facts. At the same time, we cannot wait for every question about consciousness to be answered before protecting people. Software does not need to be conscious to cause harm.
For me, mindful use of AI means understanding what we are using, checking important results and limiting what a system can do without human approval. It also means preparing for mistakes, misuse and attacks.
AI should help humanity. That must include protecting individual people, rather than treating harm to some as an acceptable price for benefits to others.
For business leaders, this means looking both at AI used inside the organisation and threats coming from outside it. Useful questions include:
- Which services or suppliers could bring our work to a stop?
- Who can quickly contain an attack and remove unwanted access?
- Can we restore our systems and check that the information is correct?
- How long could we keep paying staff and suppliers during a serious disruption?
- Have we practised our response with the organisations we depend on?
Some cooperation already exists. In 2024, the US AI Safety Institute announced testing agreements with OpenAI and Anthropic. That is a useful foundation, but it does not show that essential services around the world are ready. NIST announcement
I hope AI companies are working closely with governments, hospitals, banks, energy providers, telecommunications companies and other essential services. Where those relationships are missing, they need to be built.
Meetings and workgroups should lead to clear responsibilities, shared warnings, funded improvements and joint practice exercises. The people who would handle a real incident need to be involved.
With development moving this quickly, I do not think we should wait for complete certainty before taking these steps. We can use AI, study its benefits and ask difficult ethical questions while also preparing for things to go wrong.
Hugging Face shows that someone else’s AI can become your organisation’s problem. The next major incident may not leave time to build a response from scratch. We should be building and testing it now.