There has been a tidal wave of interest and investment in AI since the public release of ChatGPT in November 2022 by OpenAI. Over half of all global venture capital investment in 2024 was in AI-focused companies. Application areas have ranged from mundane to exotic, across a spectrum of industries. It has been used from customer service chatbots to software coding to medical imaging, from personalised marketing to graphic design to finance. Some application areas are more suited to this technology than others, and there are a number of issues, but there is a lot of potential in the technology. Amidst all the excitement, what are the actual costs of running a generative AI application? What has the return on investment for enterprises deploying it looked like so far?
To start with the costs of deploying a generative AI application in a company, there are a number of aspects to consider. As well as general project costs like project management and staffing, you need to consider the cost of data collection, data labelling and model training if needed, cloud infrastructure processing costs, integration with existing applications, deployment, compliance and monitoring, ongoing maintenance and running costs. If you simply use a publicly available large language model (LLM) rather than training your own (which can be prohibitive; Chat GPT4 cost an estimated $40-$80 million to train), then you need to take into account licensing costs. Companies like Open AI, Anthropic and Google license to enterprises based on the model being licensed, usage volume, and either a subscription or a pay-per-token model. For a small enterprise making limited demands of an LLM the costs may be modest, but for a large enterprise considering large-scale deployment, the licensing costs may be millions of dollars. One survey found that 28% of enterprises were planning to train their own LLM on private clouds or on-premises.
One key thing to consider is the ongoing cost of using LLMs, particularly if they are deployed in customer-facing applications, where the usage may be unpredictable. Each time that a customer uses your chatbot then it costs money, whether in license costs to the LLM provider, or in cloud usage bills, or both. It is important that companies carefully monitor the ongoing running costs of LLM deployments, or else they may get hit with unexpectedly high bills. One company that found out the hard way about cloud costs was software company 37signals, which faced an unexpectedly high $3 million cloud bill for its proprietary project management tool.
Even high costs may be perfectly justified if there is sufficient return on investment, but is there? Measuring the return on investment of technology projects is sketchy at the best of times. KPMG reckon that just 15% of companies are measuring ROI for AI projects. There are certainly some examples of impressive results from generative AI projects from those that actually bother to measure them. Paypal claimed an 11% reduction in losses through better risk management from their AI-driven cybersecurity project, though this was using deep learning models and not generative AI.
One survey found that large enterprises are spending an average of $2.6 million on their largest generative AI use case. A Q1 2025 Bain report found that satisfaction was high, with 40% of those surveyed claiming that AI had improved their business results, with AI budgets in this particular survey averaging $10 million, a doubling over February 2024. Other surveys differ, with a large McKinsey study of 1,491 participants finding that although 78% of companies now use AI in at least one business function, just 1% of companies described their generative AI rollouts as “mature”. In the same survey, over 80 percent of respondents say their organizations aren’t seeing a tangible impact on enterprise-level profitability from their use of generative AI. A separate study of 1,006 participants found that 46% of AI projects are scrapped between proof of concept and adoption. Similarly, 46% of respondents in that survey reported that not a single enterprise objective had seen a “strong positive impact” from that investment. The picture at present, therefore, seems distinctly mixed. This also does not take into account the reputation risk (and costs) of generative AI projects that misbehave or fail due to hallucinations, which may even result in litigation. This risk is all too real, as companies like McDonalds and Air Canada have discovered.
The above discussion ignores the costs of actually training and running LLMs from a vendor perspective. Demand has seen large-scale investment in data centres, which use prodigious amounts of electricity and water for cooling. Each query to an LLM consumes around ten times the resources of a traditional search query. The power requirements of US data centres doubled from 2022 to 2023, driven by the processing needs of LLMs. At 460 terawatts in 2022, global data centre electricity usage was already almost that of all of France. This realisation of environmental impact is gradually dawning on governments and international agencies. There are some winners, however. NVIDIA, the dominant provider of high-performance graphics processing unit (GPU) chips needed for the vector processing used by LLMs, was the most valuable company on the planet at the time of writing (July 2025), with a market cap of $3.3 trillion, more than Apple and Microsoft.
In conclusion, once you start to deploy an LLM in your enterprise, it needs to be carefully monitored and managed just like any other software application. To avoid runaway costs, the usage levels of the LLMs must be carefully monitored, measured and controlled. Above all, it is important that, like any IT project, a careful return on investment approach should be taken in order to demonstrate that the project is delivering the value that was expected and promised. A worryingly high proportion of companies appear not to be applying the usual level of rigour to measuring the return on investment of their AI projects, and such neglect may have unpleasant consequences down the line. The bills for licensing and cloud provider usage may quickly mount up if usage is not properly controlled.
