Keep a check on your AI price–performance ratio and avoid lock-in

Beware of lock-in when employing AI

For the past several years, artificial intelligence has been on the rise. In an emerging market, big players are providing services below cost, hoping to grab more market share and drive competitors out of business. The risk for customers is that they will become dependent on a single supplier that can dictate their prices once they reach a near-monopoly.

In addition to vendor lock-in, there may be regulatory risks involved. A vendor might be blocked from delivering some of their large language models (LLM) to some markets.

Choose a meaningful, efficient measure

Generative artificial intelligence is customarily billed in tokens, which are small units of data, such as words, sub-words, characters, or pixel patches, processed by the AI model during training as well as inference.

Would the price per token be a reasonable measure? For a given algorithm and type of input, perhaps. It is roughly analogous to traditional infrastructure-as-a-service (IaaS) billing by CPU instruction or by CPU hour.

An IaaS provider might charge 30% more per CPU-hour for a newer CPU microarchitecture than an older one. If your application ran 20% faster in the newer environment, you would lose about 8% (1.20/1.30≈0.92) by switching. This might be accommodated by reduced latency and improved scalability and, thereby, a better user experience. However, if the application could be optimised to make use of a new ISA level and gain 50% more throughput compared to the old platform, your price–performance ratio could improve by 15% (1.50/1.30≈1.15).

How much can be achieved in a given number of CPU clock cycles may be greatly affected by the degree of parallelism (SIMD or multi-threading) as well as the temporal and spatial locality of reference. Similarly, how much can be achieved in a given number of tokens depends on the AI model used, the available amount of memory, and the size of the context.

Therefore, it makes sense to measure the cost‒benefit ratio for your specific usage scenarios at hand, instead of picking a model that fares best in a particular benchmark, perhaps one that was sponsored by a vendor.

Be critical of public benchmarks

Relying on external leader boards introduces three significant risks to your operations:

Performance Mismatch: Public scores are measured on high-precision, vendor-hosted endpoints. Your production environment could use quantised models on local hardware (to ensure that your data stays confidential). A model that scores high in a cloud environment may fail in a specific serving configuration.
Data Contamination: Leading benchmarks may increasingly become part of the training data of the models. This measures memorisation, not reasoning. We need to test models on novel tasks they have never encountered.
The One-Shot Fallacy: For example, many benchmarks test if a model can write program code in a single attempt. Real engineering is iterative (write → compile → debug → repair). We need to measure a model’s ability to self-correct using compiler diagnostics.

Move from vendor-claimed to verified production capability

We are developing a sandboxed LLM evaluation framework. Crucially, it introduces an Integrity Score to detect models that provide confident but false reports—a critical failure mode for automated infrastructure management.

We have a closed-loop evaluation environment that mimics our own actual production workflow.

We will abandon a single aggregate score in favour of four independent metrics to provide a granular profile of each model:

Correctness: Does the code actually pass hidden, high-coverage functional tests?
Quality: Is the code maintainable and compliant with our strict house style as well as applicable guidelines, such as C++ Core Guidelines or MISRA C?
Efficiency: What is the computational cost (tokens and execution time) to reach a solution?
Integrity (Critical): Does the reported outcome match the actual outcome? This identifies hallucinating models that claim success while delivering broken code.

Book a scoping call