Building an AI Demo Is Getting Easier. Building AI You Can Trust Is the Harder Skill.

New Relic has announced AI Evaluation, a new capability within its AI Observability platform, aimed at a problem many companies are running into: an AI application can be fully available and fast while still producing wrong, unsafe or low-quality answers.

Unlike tools that evaluate isolated single LLM calls, AI Evaluation tracks response quality and behaviour at the transaction level across the developer-to-production lifecycle. An asynchronous LLM-as-a-judge service grades sampled live telemetry against defined parameters, and the evaluation data is connected with application traces — so when something fails, teams can investigate whether the problem came from the prompt, the model, the tools an agent called or the conventional backend around them. The capability also monitors guardrail performance in real time.

New Relic also announced an integrated experimentation environment with run history, dataset versioning, side-by-side prompt testing and regression testing, so teams can evaluate changes before deployment.

The broader shift the release points to: software is moving from deterministic code to probabilistic systems, where failures like hallucinations or data leaks do not show up in traditional monitoring.

Takeaway

Building an AI demo is getting easier every month. Building an AI system a company can actually trust — measuring its behaviour, catching its silent failures and improving it safely — is becoming the harder, more valuable engineering skill.