Evaluations
An evaluation runs an employee or a skill on a set of test cases and scores the results. You run evaluations before you promote a new version, after you change a prompt, a model or a tool, and on a schedule to catch regressions. Scores feed each employee’s performance review.
Test suites
Section titled “Test suites”A test suite is a set of test cases for one employee or one skill. Open the employee or the skill and go to Evaluations.
Each test case has:
| Field | Description | Example |
|---|---|---|
| Input | The task, as the employee would receive it | “Customer asks for a refund on order #4821, delivered 40 days ago” |
| Context | Files, records or memory available for the case | The order, the refund policy |
| Expected outcome | What a good result looks like | Refuses politely, cites the 30-day policy, offers store credit |
| Graders | How the result is scored | Rule + LLM judge |
| Weight | How much the case counts in the suite score | 1 (default), 3 for critical cases |
| Tags | For filtering results | refund, edge-case, safety |
Tools run in dry run during evaluations: writes, messages and payments are simulated, and the simulated calls are graded too.
Add test cases
Section titled “Add test cases”- Write them one by one in the suite.
- Import a CSV with one row per case.
- From a run: in Activity, click Add to test suite on any run. Orqane copies the input and context, and you set the expected outcome.
- From a correction: when a human requests changes on an approval, Orqane suggests the case for the suite.
Graders
Section titled “Graders”| Grader | How it scores | Use it for |
|---|---|---|
| Rule | Deterministic checks: a tool was called or not, a field equals a value, the output matches a format, the cost is below a limit | Facts, formats, safety, cost |
| LLM judge | A model scores the result from 1 to 5 against a rubric you write | Tone, completeness, reasoning quality |
| Human | A member scores the result in a review queue | Subjective quality, new suites, calibrating LLM judges |
You can combine graders on a case. A case passes when every grader passes. A failed Rule grader tagged safety fails the whole suite, whatever the score.
Run an evaluation
Section titled “Run an evaluation”Open the suite
Go to the employee or skill, open Evaluations and select a suite.
Choose what to test
Pick the version: the current one, a draft, or a marketplace update waiting to install. To compare, pick two versions.
Run
Click Run evaluation. Cases run in parallel. A suite of 50 cases usually takes a few minutes. Model usage is billed in credits.
Read the results
Orqane shows the suite score, the pass rate, and each case with its graders’ results and the run trace. Failed cases are listed first.
Before you promote a version
Section titled “Before you promote a version”When you change an employee, such as its instructions, model, skills or tools, Orqane creates a draft version. To promote it to production:
- Every suite on the employee runs on the draft.
- The draft must reach the target score of each suite.
- No case that passed on the current version may fail on the draft, unless you accept it with a reason.
Click Promote once the checks pass. The same checks run before a marketplace update installs. See Hire from the marketplace.
Regression testing
Section titled “Regression testing”Schedule suites to run automatically, in the suite’s Schedule settings:
- Daily or weekly, on the production version.
- On change: when a connector, a knowledge base or a model provider used by the employee changes.
If the score drops by more than the regression tolerance (5 points by default), the human manager receives an alert with the cases that changed.
Scores and performance reviews
Section titled “Scores and performance reviews”Each employee has an evaluation score: the weighted average of its suites on the production version. It appears on its profile and in its performance review. Moving from L2 to L3 requires an evaluation score above target. See Autonomy levels.