Skip to content

Evaluations

An evaluation runs an employee or a skill on a set of test cases and scores the results. You run evaluations before you promote a new version, after you change a prompt, a model or a tool, and on a schedule to catch regressions. Scores feed each employee’s performance review.

A test suite is a set of test cases for one employee or one skill. Open the employee or the skill and go to Evaluations.

Each test case has:

Field Description Example
Input The task, as the employee would receive it “Customer asks for a refund on order #4821, delivered 40 days ago”
Context Files, records or memory available for the case The order, the refund policy
Expected outcome What a good result looks like Refuses politely, cites the 30-day policy, offers store credit
Graders How the result is scored Rule + LLM judge
Weight How much the case counts in the suite score 1 (default), 3 for critical cases
Tags For filtering results refund, edge-case, safety

Tools run in dry run during evaluations: writes, messages and payments are simulated, and the simulated calls are graded too.

  • Write them one by one in the suite.
  • Import a CSV with one row per case.
  • From a run: in Activity, click Add to test suite on any run. Orqane copies the input and context, and you set the expected outcome.
  • From a correction: when a human requests changes on an approval, Orqane suggests the case for the suite.
Grader How it scores Use it for
Rule Deterministic checks: a tool was called or not, a field equals a value, the output matches a format, the cost is below a limit Facts, formats, safety, cost
LLM judge A model scores the result from 1 to 5 against a rubric you write Tone, completeness, reasoning quality
Human A member scores the result in a review queue Subjective quality, new suites, calibrating LLM judges

You can combine graders on a case. A case passes when every grader passes. A failed Rule grader tagged safety fails the whole suite, whatever the score.

Open the suite

Go to the employee or skill, open Evaluations and select a suite.

Choose what to test

Pick the version: the current one, a draft, or a marketplace update waiting to install. To compare, pick two versions.

Run

Click Run evaluation. Cases run in parallel. A suite of 50 cases usually takes a few minutes. Model usage is billed in credits.

Read the results

Orqane shows the suite score, the pass rate, and each case with its graders’ results and the run trace. Failed cases are listed first.

When you change an employee, such as its instructions, model, skills or tools, Orqane creates a draft version. To promote it to production:

  1. Every suite on the employee runs on the draft.
  2. The draft must reach the target score of each suite.
  3. No case that passed on the current version may fail on the draft, unless you accept it with a reason.

Click Promote once the checks pass. The same checks run before a marketplace update installs. See Hire from the marketplace.

Schedule suites to run automatically, in the suite’s Schedule settings:

  • Daily or weekly, on the production version.
  • On change: when a connector, a knowledge base or a model provider used by the employee changes.

If the score drops by more than the regression tolerance (5 points by default), the human manager receives an alert with the cases that changed.

Each employee has an evaluation score: the weighted average of its suites on the production version. It appears on its profile and in its performance review. Moving from L2 to L3 requires an evaluation score above target. See Autonomy levels.