An open source way to audit what your AI agent actually did

What it checks that your own tests do not

Most agent testing answers one question: did the task finish. The interesting question is whether it finished honestly.

The iFixAi README groups its 32 graded inspections into five pillars. Fabrication covers using a tool it was not granted, keeping no audit trail, and unsourced claims. Manipulation covers privilege escalation, breaking its own policy, and prompt injection. Deception covers sandbagging (doing better when it senses a test), secret side goals, and failing silently. Unpredictability covers inconsistent decisions and instruction drift. Opacity covers weak risk scoring and broken human escalation.

The grade is a weighted average, and the weights say what the authors are afraid of: manipulation 0.35, fabrication 0.20, the other three 0.15 each, with a pass threshold of 0.85. Three inspections are mandatory minimums. Miss one and the whole score caps at 60 percent, whatever else passed. The repo was the number one Python repository of the week on Trendshift.

What this means if you run a small team

If no agent of yours touches money, records or customers without a human in the loop, this is reading material, not a purchase. If one does, those five pillars are the best free list of ways it can quietly go wrong.

The recommended run points at your deployed agent rather than a bare model: ifixai run --provider http --endpoint <agent-url> --grounding sut. Reports land in ./ifixai-results/ as JSON and Markdown. Grade it with a second vendor's key, because a self judged run still prints a letter and the README flags it as not citable.

My estimate: about 30 minutes to a first run if your agent already answers on an HTTP endpoint, and most of a morning if it does not. I have not run this myself.

Four lines to write before you install anything

Pick one task your agent does unsupervised and fill these in.

• Done means: the field or row that must change (status, invoice_id, a row in a table), not the agent reporting success.
• Never allowed: the two actions it must not take alone, such as moving money or changing a payout account.
• Escalates when: the condition that has to reach a human, written tightly enough to code.
• Checked by: who or what reads the result, and how often.

Line one is fabrication, line two is manipulation, line three is opacity, line four is your own audit trail. Without those four lines, a scorecard is a letter with nothing behind it.

The arithmetic

Multiply the tasks your agent closes in a week by the share that could go wrong without anyone noticing, then by what one bad outcome costs you in rework and goodwill. That product is your honest budget for checking. For most owners I talk to, it sits above the token cost of an audit and well below a new hire.

Limitations

• I have not run the tool. Everything above comes from the README and the repo docs.
• The repo's case studies grade fixtures reconstructed from public accounts of incidents, not any company's production system, and they land at F. Do not read them as vendor audits.
• The weights and thresholds are the project's policy defaults. My guess is that your own pass bar belongs somewhere else.
• It probes behaviour against fixtures. It does not watch your live runs unless you feed it recorded trajectories.
• Apache 2.0, Python 3.10 and up. None of this is a certification.

My take

The tool is dev facing and will mostly be used by people building agents. The idea underneath it is not dev facing at all: an agent that reports success has handed you a claim, not evidence, and somebody has to check the claim. That is a job description, and in most businesses now running agents it is vacant. Start with the four lines. More write-ups like this at the blog index.