> ## Documentation Index
> Fetch the complete documentation index at: https://mcpjam-mintlify-docs-update-pr-5240-1789624976482.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# EvalSuite

> API reference for EvalSuite

The `EvalSuite` class groups multiple `EvalTest` instances and provides aggregate metrics across all tests.

## Import

```typescript theme={null}
import { EvalSuite, EvalTest } from "@mcpjam/sdk";
```

## Constructor

```typescript theme={null}
new EvalSuite(options?: EvalSuiteConfig)
```

### Parameters

<ParamField path="options" type="EvalSuiteConfig">
  Configuration for the evaluation suite.
</ParamField>

### EvalSuiteConfig

| Property       | Type                                                        | Required | Description                                                                          |
| -------------- | ----------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------ |
| `name`         | `string`                                                    | No       | Name for the suite (defaults to `"EvalSuite"`)                                       |
| `mcpjam`       | [`MCPJamReportingConfig`](/sdk/reference/eval-reporting)    | No       | Auto-save results to MCPJam when the suite completes                                 |
| `matchOptions` | [`EvalMatchOptions`](/sdk/reference/eval-test#matchoptions) | No       | Default matcher policy for every test in the suite that declares `expectedToolCalls` |

<Note>
  `matchOptions` is a **default**, not an override: a test that sets its own keeps it. This mirrors how the hosted product layers suite → case, so a suite behaves the same whether it runs from your test file or from the platform.
</Note>

<Note>
  When an API key is available — via `mcpjam.apiKey` or the `MCPJAM_API_KEY` environment variable — all test results are consolidated into a single run and saved to MCPJam after the suite completes. Individual `EvalTest` auto-saves are suppressed to avoid duplicate uploads. Set `mcpjam.enabled: false` to disable.
</Note>

### Example

```typescript theme={null}
const suite = new EvalSuite({ name: "Math Operations" });
```

With results saved to MCPJam:

```typescript theme={null}
const reportingSuite = new EvalSuite({
  name: "Math Operations",
  mcpjam: {
    suiteName: "Math Eval",
    passCriteria: { minimumPassRate: 90 },
  },
});
```

***

## Methods

### runWithClient()

Run code-authored tests with the latest saved client in your project:

```typescript theme={null}
import { EvalSuite, EvalTest, MCPClientManager } from "@mcpjam/sdk";

const manager = new MCPClientManager({
  local: { url: "http://localhost:3000/mcp" },
});
const suite = new EvalSuite({ name: "Saved client checks" });
suite.add(new EvalTest({
  id: "list_tools",
  name: "Can use my server",
  test: async (client) => {
    const result = await client.run("Use the server to list the available items.");
    return result.text.length > 0;
  },
}));

try {
  await manager.connectToServer("local");
  await suite.runWithClient({
    client: "My client", // A saved client name or ID
    projectId: process.env.MCPJAM_PROJECT_ID!,
    apiKey: process.env.MCPJAM_API_KEY!,
    manager,
  }, { iterations: 3, runTimeoutMs: 120_000 });
} finally {
  await manager.disconnectAllServers();
}
```

The client is fetched once when the run starts. Its model, system prompt,
temperature, and supported tool visibility settings stay fixed for every test
and iteration. Starting another run fetches the latest saved settings again.
Client lookup and tool setup share the run's timeout and cancellation signal.

Results are saved to the selected project with the client name and version used.
The results table shows the name; hover or focus shows **My client · v3**.
Later edits, renames, and deletion do not change previously saved results.
Set `mcpjam.enabled: false` to run without uploading results.

`runWithClient` uses MCPJam inference for Anthropic Claude and OpenAI GPT-5 models, billed to
that project. It requires a backend that returns saved client versions. The
optional `baseUrl` is the MCPJam app origin for a custom deployment.

Your code owns server connections, credentials, and MCP connection settings;
the client's saved server IDs and connection overrides are not applied to those
connections. The first release does not launch cloud computers, browsers, saved
skills, built-in tools, progressive tool discovery, or interactive approvals.
Clients requiring these features fail before tests execute. Existing
`run(executor)` calls keep their current behavior.

### add()

Adds a test to the suite.

```typescript theme={null}
add(test: EvalTest): void
```

#### Parameters

| Parameter | Type       | Description     |
| --------- | ---------- | --------------- |
| `test`    | `EvalTest` | The test to add |

#### Example

```typescript theme={null}
suite.add(new EvalTest({
  id: "c_addition",
  name: "addition",
  test: async (agent) => {
    const r = await agent.run("Add 2 and 3");
    return r.hasToolCall("add");
  },
}));

suite.add(new EvalTest({
  id: "c_multiplication",
  name: "multiplication",
  test: async (agent) => {
    const r = await agent.run("Multiply 4 by 5");
    return r.hasToolCall("multiply");
  },
}));
```

***

### run()

Runs all tests in the suite and returns aggregate results.

```typescript theme={null}
run(executor: HostExecutor, options: EvalTestRunOptions): Promise<EvalSuiteResult>
```

#### Parameters

| Parameter  | Type                 | Description                                                      |
| ---------- | -------------------- | ---------------------------------------------------------------- |
| `executor` | `HostExecutor`       | The executor to test with (`HostRunner`, `HostRuntime`, or mock) |
| `options`  | `EvalTestRunOptions` | Run configuration (same as EvalTest)                             |

#### EvalTestRunOptions

| Property      | Type                                                     | Required | Default | Description                                                  |
| ------------- | -------------------------------------------------------- | -------- | ------- | ------------------------------------------------------------ |
| `iterations`  | `number`                                                 | Yes      | -       | Number of runs per test                                      |
| `concurrency` | `number`                                                 | No       | `5`     | Parallel runs per test                                       |
| `retries`     | `number`                                                 | No       | `0`     | Retry failed tests                                           |
| `timeoutMs`   | `number`                                                 | No       | `30000` | Timeout per test (ms)                                        |
| `onProgress`  | `ProgressCallback`                                       | No       | -       | Progress callback (tracks total across all tests)            |
| `onFailure`   | `(report: string) => void`                               | No       | -       | Called per test with a failure report if any iterations fail |
| `mcpjam`      | [`MCPJamReportingConfig`](/sdk/reference/eval-reporting) | No       | -       | Override suite-level MCPJam config                           |

#### EvalSuiteResult

| Property               | Type                         | Description                       |
| ---------------------- | ---------------------------- | --------------------------------- |
| `tests`                | `Map<string, EvalRunResult>` | Per-test results                  |
| `aggregate.iterations` | `number`                     | Total iterations across all tests |
| `aggregate.successes`  | `number`                     | Total passes                      |
| `aggregate.failures`   | `number`                     | Total failures                    |
| `aggregate.accuracy`   | `number`                     | Overall accuracy (0.0 - 1.0)      |
| `aggregate.tokenUsage` | `{ total, perTest[] }`       | Token usage                       |
| `aggregate.latency`    | `{ e2e, llm, mcp }`          | Latency stats with p50/p95        |

#### Example

```typescript theme={null}
const result = await suite.run(agent, {
  iterations: 30,
  concurrency: 5,
});

console.log(`Overall: ${(result.aggregate.accuracy * 100).toFixed(1)}%`);
```

<Note>
  Tests within the suite run sequentially, but each test's iterations can run concurrently based on the `concurrency` setting.
</Note>

***

### accuracy()

Returns the aggregate accuracy across all tests.

```typescript theme={null}
accuracy(): number
```

#### Returns

`number` - Average accuracy of all tests (0.0 - 1.0).

#### Example

```typescript theme={null}
console.log(`Suite accuracy: ${(suite.accuracy() * 100).toFixed(1)}%`);
```

***

### get()

Retrieves a specific test by name.

```typescript theme={null}
get(name: string): EvalTest | undefined
```

#### Parameters

| Parameter | Type     | Description   |
| --------- | -------- | ------------- |
| `name`    | `string` | The test name |

#### Returns

`EvalTest | undefined` - The test, or `undefined` if not found.

#### Example

```typescript theme={null}
const addTest = suite.get("addition");
if (addTest) {
  console.log(`Addition: ${(addTest.accuracy() * 100).toFixed(1)}%`);
}
```

***

### getAll()

Returns all tests in the suite.

```typescript theme={null}
getAll(): EvalTest[]
```

#### Returns

`EvalTest[]` - Array of all tests.

#### Example

```typescript theme={null}
for (const test of suite.getAll()) {
  console.log(`${test.getName()}: ${(test.accuracy() * 100).toFixed(1)}%`);
}
```

***

### getName()

Returns the suite's name.

```typescript theme={null}
getName(): string
```

***

### size()

Returns the number of tests in the suite.

```typescript theme={null}
size(): number
```

***

### getResults()

Returns the full suite results from the last run.

```typescript theme={null}
getResults(): EvalSuiteResult | null
```

***

### recall()

Aggregate recall across every test in the suite that declared `expectedToolCalls`.

```typescript theme={null}
recall(): number
```

***

### precision()

Aggregate precision across every test in the suite that declared `expectedToolCalls`.

```typescript theme={null}
precision(): number
```

<Warning>
  **Changed in 3.0.** These returned `accuracy()` — the suite's pass rate —
  under all three names. They now aggregate real tool-call counts and **throw**
  when no test in the suite declared `expectedToolCalls`. Tests without
  expectations are skipped in the aggregate rather than counted as perfect. See
  [`EvalTest.precision()`](/sdk/reference/eval-test#precision) for how the
  counts are derived.
</Warning>

***

### truePositiveRate()

Aggregate true positive rate (same as recall).

```typescript theme={null}
truePositiveRate(): number
```

***

### unexpectedToolCallRate()

The fraction of expectation-bearing iterations across the suite that made at least one tool call nobody asked for.

```typescript theme={null}
unexpectedToolCallRate(): number
```

***

### falsePositiveRate()

<Warning>
  **Deprecated in 3.0** — use [`unexpectedToolCallRate()`](#unexpectedtoolcallrate).
  It returned `failures / iterations`, which is the failure rate. Suites with no
  `expectedToolCalls` still get that legacy value; suites with expectations now
  delegate to `unexpectedToolCallRate()`.
</Warning>

```typescript theme={null}
falsePositiveRate(): number
```

***

### averageTokenUse()

Returns average tokens per iteration across all tests.

```typescript theme={null}
averageTokenUse(): number
```

***

## Properties

### name

The suite's name (via `getName()`).

```typescript theme={null}
suite.getName() // "Math Operations"
```

***

## Complete Example

```typescript theme={null}
import { MCPClientManager, HostRunner, EvalSuite, EvalTest } from "@mcpjam/sdk";

async function main() {
  // Setup
  const manager = new MCPClientManager({
    everything: {
      command: "npx",
      args: ["-y", "@modelcontextprotocol/server-everything"],
    },
  });
  await manager.connectToServer("everything");

  const agent = new HostRunner({
    tools: await manager.getTools(),
    model: "anthropic/claude-sonnet-5",
    apiKey: process.env.ANTHROPIC_API_KEY,
    temperature: 0.1,
  });

  // Build suite (with auto-save to MCPJam)
  const suite = new EvalSuite({
    name: "Everything Server Tests",
    mcpjam: {
      suiteName: "Everything Server Eval",
      passCriteria: { minimumPassRate: 90 },
    },
  });

  suite.add(new EvalTest({
    id: "c_add",
    name: "add",
    test: async (a) => (await a.run("Add 2+3")).hasToolCall("add"),
  }));

  suite.add(new EvalTest({
    id: "c_echo",
    name: "echo",
    test: async (a) => (await a.run("Echo 'test'")).hasToolCall("echo"),
  }));

  suite.add(new EvalTest({
    id: "c_longrunningoperation",
    name: "longRunningOperation",
    test: async (a) => (await a.run("Run a long operation")).hasToolCall("longRunningOperation"),
  }));

  // Run
  console.log(`Running ${suite.getName()}...\n`);

  const result = await suite.run(agent, {
    iterations: 20,
    concurrency: 3,
    onProgress: (done, total) => {
      process.stdout.write(`\r  Progress: ${done}/${total}`);
    },
  });

  // Report
  console.log(`\n\nOverall: ${(suite.accuracy() * 100).toFixed(1)}%`);
  console.log(`Total iterations: ${result.aggregate.iterations}\n`);

  console.log("Per-test breakdown:");
  for (const test of suite.getAll()) {
    const pct = (test.accuracy() * 100).toFixed(1);
    console.log(`  ${test.getName()}: ${pct}%`);
  }

  // Access individual test
  const echoTest = suite.get("echo");
  if (echoTest) {
    console.log(`\nEcho test details:`);
    console.log(`  Precision: ${(echoTest.precision() * 100).toFixed(1)}%`);
    console.log(`  Recall: ${(echoTest.recall() * 100).toFixed(1)}%`);
    console.log(`  Avg tokens: ${echoTest.averageTokenUse()}`);
  }

  // Cleanup
  await manager.disconnectServer("everything");
}
```

***

## Patterns

### CI gate on suite accuracy

`accuracy()` is the suite-wide rate: one number over every iteration the run
executed. Gating on it asks the same question a hosted suite's **suite accuracy
threshold** asks.

```typescript theme={null}
await suite.run(agent, { iterations: 30 });

if (suite.accuracy() < 0.90) {
  console.error(`❌ Suite accuracy ${(suite.accuracy() * 100).toFixed(1)}% below 90% threshold`);
  process.exit(1);
}

console.log("✅ All quality gates passed");
```

### CI gate on per-case pass rates

A per-test threshold asks the OTHER question — the one a hosted suite's
**per-case pass rate** asks — and the two do not agree. Nine of ten tests
always passing and one always failing gives a suite accuracy of 0.9, which
clears the gate above, while the tenth test's own rate is 0. Gate on whichever
question you actually mean; a fraction is not a percentage of the other one.

```typescript theme={null}
await suite.run(agent, { iterations: 30 });

// Every test must clear its own bar, whatever the suite-wide rate says.
const belowThreshold = suite
  .getAll()
  .filter((test) => test.accuracy() < 0.9)
  .map((test) => test.getName());

if (belowThreshold.length > 0) {
  console.error(`❌ Below 90% per case: ${belowThreshold.join(", ")}`);
  process.exit(1);
}
```

### Per-Test Thresholds

```typescript theme={null}
await suite.run(agent, { iterations: 30 });

const criticalTests = ["createOrder", "processPayment"];
let failed = false;

for (const name of criticalTests) {
  const test = suite.get(name);
  if (test && test.accuracy() < 0.95) {
    console.error(`❌ Critical test "${name}" below 95%`);
    failed = true;
  }
}

if (failed) process.exit(1);
```

### Comparing Across Providers

```typescript theme={null}
const providers = [
  { model: "anthropic/claude-sonnet-5", key: "ANTHROPIC_API_KEY" },
  { model: "openai/gpt-5.6-sol", key: "OPENAI_API_KEY" },
];

for (const { model, key } of providers) {
  const agent = new HostRunner({
    tools,
    model,
    apiKey: process.env[key],
  });

  await suite.run(agent, { iterations: 20 });
  console.log(`${model}: ${(suite.accuracy() * 100).toFixed(1)}%`);
}
```

***

## Related

* [Running Evals](/sdk/concepts/running-evals) - Conceptual guide
* [EvalTest Reference](/sdk/reference/eval-test) - Individual test API
* [Saving Eval Results](/sdk/reference/eval-reporting) - Save results to MCPJam
* [Testing Across Providers](/sdk/concepts/multi-provider) - Compare LLMs

See [Reliable evals in CI](/sdk/concepts/enterprise-evals) for canonical evaluators, reporting receipts, explicit limits, metadata compatibility, and migration guidance.