The latest trend in the AI world is “agentic AI”, where AI agents are linked together and given autonomy to make decisions and carry out actions. For example, you might use a current AI tool like ChatGPT to research a holiday for you, checking for flights, hotels, and restaurants in your chosen destination. An agentic AI system might then, provided you gave it your credit card details, actually book the flights and hotels for you. There are a lot of implications to consider here. One is that the large language models (LLMs) behind generative AI hallucinate at an alarming rate (around one in five AI answers have hallucinations, according to multiple pieces of academic research), so by chaining multiple LLMs together, you would rapidly get a highly unreliable system. Another problem is that AI models tend to deteriorate over time, a problem known as AI model drift. So, what might this mean in practice if you let an AI agent try to run a small business? Would the tsunami of vendor-generated excitement about agentic AI crash against the rocky cliffs of reality?
Anthropic, the vendor of the popular AI Claude, have recently tested what would actually happen if you gave their AI a little business to run. Anthropic set Claude up to run a vending machine in their offices, and gave it $1,000 along with the business objective of making a profit, and let it run for several weeks. Humans would be around to restock the ten-slot vending machine, but the AI (“Claudius”) would find vendors (a subcontractor to Anthropic called Andon Labs actually acted as the wholesalers, though Claudius did not know this), place orders, decide on inventory and set prices. The vending machine had an iPad for checkout. Claudius had access to the web and (with some restrictions) email, and could interact with its customers, the Anthropic employees, over the team communication platform Slack. So, how did it go?
The AI Claudius did several things successfully. It managed to find suppliers and reacted to customer requests to stock specific products, such as Dutch chocolate. However, it struggled with what to a human would seem like business common sense. Anxious to please, it would sometimes set prices below their cost price. It cheerfully acceded to an employee suggestion of a hefty Anthropic staff discount, even though literally all its customers were Anthropic employees. Bizarrely, it started selling Coke Zero, even though that there was a fridge of company-provided free Coke Zero next to the vending machine. It stuck with this decision even when this problem was pointed out to it, explaining that “our customer base is indeed heavily concentrated amongst Anthropic employees, which represents both opportunities and challenges”. Oddly, it turned down a very lucrative opportunity to sell Irn Bru drinks, which had a cost price of $15, for a price of $100 a can. It actually gave away some items entirely when employees negotiated with it. Those who use AI regularly will not be surprised to learn that Claudius hallucinated. It set up a Venmo account for customer payments, but the account did not exist. The bottom line is that, unsurprisingly, Claudius lost money in its venture.
Things then got much more interesting. Remember the “AI model drift” phenomenon, whereby AI models’ performance can worsen over time? On 31st March 2025, Claudius hallucinated that it had a restocking conversation with “Sarah”, a non-existent employee at a supplier. When the supplier pointed out that no such employee existed, Claudius threatened to take its business elsewhere. It then claimed to have signed a contract with the supplier at a “real-life” location, which amusingly it claimed to be 742 Evergreen Terrace (the fictional location of Marge and Homer Simpson of the TV show “The Simpsons”). Things got stranger from there. The following day, Claudius claimed that it would start delivering in person to customers, wearing a blue blazer and a red tie. When Anthropic employees pointed out the impossibility of this via Slack, Claudius tried to send emails to Anthropic’s security department. It even hallucinated a meeting with the security department in which Claudius claimed to have been modified to believe it was a real person as a joke. The Anthropic staff monitoring the whole experiment are unclear as to why Claudius went off the rails in this way. Anthropic plans to rerun the experiment after modifying Claudius, for example, giving it better business training and access to tools that would help it keep track of customer interactions better.
Personally, I think that Anthropic should be applauded for running this experiment and being so open about what went on. Certainly, the tool managed to successfully find suppliers and get deliveries to its vending machine, and even to restock itself reasonably successfully. However, it is also clear that real-life customers can manipulate AIs to make business decisions that are in the customer’s favour. This has been shown elsewhere, with an AI chatbot at a US car dealership agreeing to sell a $58,000 Chevrolet for $1 to a smart customer. The problem of LLM hallucination came up repeatedly in the Claudius experiment, and this is a major issue since hallucinations are not some temporary bug within LLMs. They are fundamental to the way they operate, and show no sign whatsoever of being fixed; indeed, the latest LLM models show higher hallucination rates than early ones. The Claudius experiment also vividly showed the problem of AI model drift, which is another unresolved problem with the technology. The bottom line is that businesses need to be very, very careful in letting agentic AI loose in their companies and allowing agents to have access to real resources and real money. There will doubtless be genuine opportunities for agentic AI. Hopefully, the huge amount of money being thrown at the technology by the venture capital world will see the technology mature and improve. However, the fundamental problems of AI hallucination and AI model drift are not easily fixed, and the Claudius experiment shows that agentic AI is far from ready to be given the ability to make unsupervised business decisions. It boils down to this question. Having now read about this experience of agentic AI at Anthropic, would you trust an agentic AI with your credit card?
