Office 11.041 Spaces Canary Wharf, 25
Cabot Square,
London, England, E14 4QZ
Your AI agent got the right answer. That doesn’t mean it worked well.
NVIDIA’s recent work on evaluating agent skills highlights a common problem in production. Even a capable model can waste steps, burn tokens on dead ends, or cycle through the wrong tools before reaching the right result.
On a demo, two agents can look identical:
Agent A: intent ? correct tool ? result
Agent B: intent ? wrong tool ? retry ? redundant call ? more tokens ? correct tool ? result
Both pass a basic success check. In production, they are completely different systems.
NVIDIA’s open-source SkillEvaluator shows why final correctness isn't enough. It separately measures correctness, effectiveness, discoverability, efficiency, and security, tracking whether the agent actually reaches the goal without wasted steps or redundant tool calls.
Across more than 300 verified skills, having the right skill raised average correctness from 46 to 87 and effectiveness from 39 to 78.
SkillEvaluator tracks token usage separately from its efficiency metric. In one example, the jetson-optimize-memory skill reduced token consumption by 76.9%. In another, cuopt-install increased it by 120.3%. Making an agent better at completing a task doesn't automatically make its execution cheaper.
From a production perspective, redundant tool calls can mean higher latency, higher costs, and more points of failure. Task completion tells you if an agent can finish the job. Its execution trajectory shows you how it got there.
In production, both matter.
At DataObrii, we evaluate agentic systems beyond task completion: tool use, retries, efficiency, failure modes, and behavior under real production constraints.
Source: NVIDIA, “Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator.”
Share This News