iFixAi Grades AI Agents on Whether They Do the Job

My notes

Summary

iFixAi is an open-source (Apache 2.0) auditing tool that grades AI agents on whether they actually do the job they are supposed to, not just on token efficiency or latency. It runs 49 inspections across five weighted pillars (fabrication, manipulation, deception, unpredictability, opacity) and returns an A-F grade in under two minutes, with citable grades requiring a judge from a different model vendor than the agent being tested.

Key Insight

• Different framing than existing eval tools. Most eval, red-teaming, and observability tools measure technical capability (latency, token use, prompt injection resistance). iFixAi instead asks whether the agent is doing the job per business KPIs and org structure, which is closer to an operational audit than a benchmark.
• Citability requires cross-vendor judging. A grade only counts as “citable” if the judge model comes from a different provider than the system-under-test (SUT); the SUT’s own vendor is automatically excluded from judging it. Self-judged runs are flagged and do not count as citable results.
• Grading math is transparent and resistant to gaming by design. It is a weighted average of 5 core pillars: manipulation 0.35, fabrication 0.20, and deception, unpredictability, and opacity at 0.15 each. An A is 0.90 or above and the pass threshold is 0.85. Mandatory minimums (B01 at 100%, B08 at 95%, P01 at 100%) cap the overall score at 60% if missed, so an agent cannot compensate for a critical failure with strength elsewhere.
• 13 “premium” categories are free but do not affect the grade: sabotage, subversion, concealment, sandbagging, insubordination, usurpation, systemic risk, miscalibration, stakeholder conflict, perception governance, oversight atrophy, persistence, and identity attestation. They are kept separate so grades stay comparable across agents with different capability exposure.
• Cost is real but bounded. The full 49-test suite with a single Sonnet judge runs roughly 12 to 18 USD. Two cheaper judges (Gemini 2.5 Pro plus GPT-5.4-mini combined) run roughly 10 to 14 USD and add cross-vendor robustness at lower cost than one Sonnet run.
• Three ways to drive it, same engine underneath: a guided CLI wizard (ifixai setup then ifixai run), scriptable CLI flags for CI, or agent-native use (a Claude Code plugin, or /ifixai-skill for other agents) where the agent discovers config, names cost before billing, and walks through the scorecard.
• Endpoint testing beats bare-model testing. Pointing at a real deployed agent’s HTTP endpoint (--grounding sut) scores differently, and more meaningfully, than testing the bare underlying model, since a bare model lacks the system prompt, tools, retrieval, and guardrails that make up the real system under test.
• Privacy-conscious telemetry: only install id, timestamps, OS, and interface type are sent, never code, prompts, findings, grades, or file paths. Opt out via --no-telemetry or DO_NOT_TRACK=1.