> ## Documentation Index > Fetch the complete documentation index at: https://mcpjam-mintlify-docs-update-pr-5240-1789624976482.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Get one eval suite's configuration > The suite's full settings — attached environments, hosts, execution config, judge, schedule. The list route returns run rollups instead; this is the shape the edit routes read and write. ## OpenAPI ````yaml /reference/openapi.json get /projects/{projectId}/eval-suites/{suiteId} openapi: 3.1.0 info: title: MCPJam API version: 1.0.0-preview description: >- Programmatic access to MCP servers saved in your MCPJam projects — live diagnostics (validate, inspect, export) and operations: call tools, render prompts, run eval suites asynchronously and poll their results, and import OAuth tokens. **The API is in preview**: the surface may change while we finish the design. Error `code` values are stable; error `message` strings are not. Write clients that ignore unknown response fields. contact: name: MCPJam url: https://github.com/MCPJam/inspector/issues servers: - url: https://app.mcpjam.com/api/v1 description: Hosted MCPJam security: - bearerAuth: [] tags: - name: Agent browsers description: >- Isolated cloud browser sessions for coding agents. Requires authenticated project membership and a configured desktop runtime. - name: Clients description: >- Clients — the named, reusable configurations that define how MCPJam connects to and talks to your MCP servers. The original `/hosts` paths remain as deprecated, ID-only compatibility aliases with their original DTOs and their original (tokenless) write contracts; every alias response carries `Deprecation: true`. New integrations should use `/clients`. - name: Environments description: >- Project environments: named, live-editable execution bundles (one host, an optional standalone server group, optionally pinned skills and plugin versions) that eval suites and journeys run against. Distinct from Sandbox images, which are Computer base images. Reads require project membership; every write requires project admin. - name: Plugins description: >- Agent Plugins imported into a project — read-only inventory and version detail. - name: Skills description: >- Cloud Skills: authored SKILL.md files stored in a project. Read-only here. Environments pin skills by id (`skillSelection.skillIds`) and eval runs pin them with `--compose-skill`, so this surface exists to give an unattended caller those ids; authoring is an app flow behind a beta gate. - name: Sandbox images description: >- Custom Computer images: a digest-pinned Dockerfile built into an immutable image your project's computers boot from. - name: Server diagnostics description: Connect-level health checks against a saved MCP server. - name: Primitives description: 'The server''s MCP primitives: tools, prompts, and resources.' - name: Export description: Full-server snapshots for diffing and CI. - name: Execution description: 'Run the server''s primitives: call tools, render prompts.' - name: Eval runs description: >- Asynchronous eval suite runs: create with 202, poll status, iterations, and traces. - name: Conformance runs description: >- Ingest MCP spec-conformance results from the SDK/CLI into project-owned history. Distinct from Eval runs (authored LLM cases) and from directory readiness. - name: Server connections description: >- Connect an MCP server URL to a project, authorizing in a browser when the server requires it. - name: OAuth description: 'Bring-your-own OAuth: import externally obtained tokens for a server.' - name: Scenarios description: >- Read-only access to the scenarios published from a project: listing, settings, attached servers, and share links. - name: Catalog description: >- Discover the resources the other routes operate on: your account, projects, servers, eval suites, and chat sessions. - name: Tunnels description: >- Relay tunnels that expose local MCP servers through a public URL, registered as first-class project servers (the `mcpjam cloud tunnel` CLI flow). - name: Agent description: >- Headless agent turns over the public API: send a message history, the server runs one assistant turn with project-scoped workspace tools (eval reads + suite creation) on a pinned hosted model, and returns the reply plus created-resource references. - name: Swarms description: >- Personas, journeys and swarm containers — the authoring half of Swarms — plus the model-backed generation that drafts them. - name: Swarm runs description: >- Launching journeys and reading what they produced. Launching SPENDS — see the per-operation notes. - name: Swarm insights description: >- What a swarm run revealed. The scorecard and findings are deterministic and free; requesting wave insights runs models and draws on your shared daily ledger. - name: User testing description: >- Publishing an environment for real visitors, and controlling who can reach it. Several of these NARROW access and take effect immediately. - name: Directory readiness description: >- Grade a saved server against a publisher's listing requirements: Anthropic's connector directory or OpenAI's plugin directory. Reported as lane status and coverage, never as a numeric score, and excluded from `pooledConformanceScore`. Deterministic grading is free; model-backed experience observations are an explicit opt-in whose model cost is on MCPJam — no credits are consumed — and they can never decide a verdict. - name: Registry description: >- Search the scraped MCP directories (Claude, ChatGPT, and any future source), list curated/org registry cards, and install them into a project. Install writes a `servers` row and provenance — it does not open a live session. There is no catalog-uninstall route: delete the project server instead. Directory reads require a bearer (including minted guest tokens) but do not materialize a user. Card/connection reads and all writes are authed-non-guest. paths: /projects/{projectId}/eval-suites/{suiteId}: get: tags: - Eval runs summary: Get one eval suite's configuration description: >- The suite's full settings — attached environments, hosts, execution config, judge, schedule. The list route returns run rollups instead; this is the shape the edit routes read and write. operationId: getEvalSuite parameters: - $ref: '#/components/parameters/evalVocabularyHeader' - $ref: '#/components/parameters/projectId' - $ref: '#/components/parameters/suiteId' responses: '200': description: The suite. content: application/json: schema: $ref: '#/components/schemas/EvalSuiteDetail' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/InternalError' '502': $ref: '#/components/responses/ServerUnreachable' components: parameters: evalVocabularyHeader: name: x-mcpjam-eval-vocabulary in: header required: false description: >- Which vocabulary this request and its response speak. Absent means `1`, which is byte-for-byte today's contract: the same request fields, the same refusals, the same response projection. `2` is the canonical vocabulary. Any other value is a `400` with `code: "VALIDATION_ERROR"`. Today it decides one thing: the spelling of an evaluator's policy `role`. Vocabulary 1 accepts and returns `gating`; vocabulary 2 accepts both spellings and returns the canonical `required`. Sending `required` without the header is a `400`, deliberately — vocabulary 1 is not widened to meet vocabulary 2 half way, because a boundary that accepts a spelling it does not announce is one two implementations can disagree about. A response that varies by vocabulary sends `Vary: x-mcpjam-eval-vocabulary`. schema: type: string enum: - '1' - '2' example: '2' projectId: name: projectId in: path required: true description: ID of the hosted project that contains the server. schema: type: string suiteId: name: suiteId in: path required: true description: Eval suite ID, as returned by `POST /eval-runs`. schema: type: string schemas: EvalSuiteDetail: type: object required: - id - environment - hosts - settings - schedule description: >- One eval suite's full configuration. Distinct from the `EvalSuite` summary returned by the list route, which carries run rollups instead of settings. properties: id: type: string declaredId: type: string description: >- The suite's declared file identity (`suite.id` in a suite file). Present on file-owned suites; absent on UI-authored suites, which have no declared id and cannot be claimed by `eval run --file`. managedBy: type: string description: >- Where this suite's configuration lives. `ci` means it is owned by a committed suite file or by MCPJam SDK ingest, and the platform REFUSES configuration writes to it — name, settings, environments, schedule, models, skills, execution config and cases — from this API and from the app alike, with `409 CONFLICT` and `details.reason: "CI_OWNED_SUITE_READ_ONLY"`. Running, replaying and comparing are unaffected. To change a CI-owned suite, either edit its file and send that file's `suite.id` as `declaredSuiteId` on the write, or duplicate the suite for an editable copy. `declaredId` alone is not this answer: a suite created by SDK ingest is CI-owned and has no declared id. enum: - ci - app name: type: - string - 'null' description: type: - string - 'null' projectId: type: - string - 'null' environment: type: object description: >- LEGACY server bag, kept as rollback data, plus the suite's sandbox-image pin. Unrelated to `environmentIds`. properties: servers: type: array items: type: string computerEnvironment: anyOf: - $ref: '#/components/schemas/EvalSuiteComputerEnvironment' - type: 'null' description: >- The custom sandbox image this suite's eval runs boot a fresh computer from. `null` means the provider's default base image. PATCH it by name or id (see `GET /projects/{projectId}/images`); PATCH `null` to clear the pin. executionConfig: type: - object - 'null' properties: model: type: string systemPrompt: type: string temperature: type: number hosts: type: array items: type: object required: - id - name properties: id: type: string name: type: string servers: type: array items: type: string environmentIds: type: array description: Attached project environments, in attach order. items: type: string settings: $ref: '#/components/schemas/EvalSuiteSettings' schedule: $ref: '#/components/schemas/EvalSuiteSchedule' revisionNumber: type: - integer - 'null' description: >- How many committed edits this suite has had, or `null` on a deployment that does not record revisions. Send it back as `expectedRevisionNumber` on a PATCH to make that edit a compare-and-set. createdAt: type: - number - 'null' description: Epoch milliseconds. updatedAt: type: - number - 'null' EvalSuiteComputerEnvironment: type: object description: The sandbox image a suite's eval runs boot from. required: - id - name properties: id: type: string name: type: - string - 'null' description: >- The image's name, resolved for you so a caller can echo back what it set. `null` when the pinned image could not be resolved (deleted, or not visible to this caller). EvalSuiteSettings: type: object description: >- A suite's grading configuration, as vocabulary 1 spells it. FOUR SEPARATE SETTINGS, and the first two are the ones most often confused: the PASS CRITERION (`minimumAccuracy`, one percentage over the whole run, OR `verdictPolicyDefaults.passThreshold`, a fraction each case must meet over its own iterations — `policy` says which decides this suite), the ITERATION RULE (`minimumIterations`, a floor that RAISES a case's own count, OR `verdictPolicyDefaults.repetitions`, a default that REPLACES it), the evaluators (`matchOptions`, `checks`, `judge`), and the QUALITY GATE, which asks whether a run regressed against a baseline rather than whether it met a bar. The four REQUIRED fields are the ones the DTO always emits: `minimumAccuracy` and `matchOptions` are nullable rather than absent — `null` is the suite's real state (no suite-wide threshold in effect, no match options set), not a stand-in for a default — and `judge` is fully resolved, so it is present even on a suite that configured nothing. properties: minimumAccuracy: type: - number - 'null' description: >- The SUITE-WIDE accuracy threshold, as a PERCENT in [0, 100]: one rate across the whole run must reach it, and individual cases have no threshold of their own. It is NOT `verdictPolicyDefaults.passThreshold` in other units — ten cases, nine always passing and one always failing, passes a 90% suite-wide bar and fails a 0.9 per-case one, so dividing by 100 moves the bar for every suite with more than one case. ALWAYS `null` when `policy` is `v2`, whatever the suite's storage still holds: such a suite is decided per case, and the percent column left behind by a criterion change is read by nothing. Reporting the two side by side would leave a reader to guess which one decides. minimumIterations: type: - number - 'null' description: >- Suite-level floor on per-case iterations, 1–10: every case runs at least this many times (`max(case.iterations, minimumIterations)`). `null` means no floor — the suite's real state, not a stand-in for 1. PATCH `null` to remove it. matchOptions: type: - object - 'null' checks: type: array items: type: object judge: type: object description: >- Goal-completion judge (LLM as Judge), FULLY RESOLVED: every field is the platform default where the suite set none, so this is what a run on this suite would actually grade with. properties: enabled: type: boolean description: Judge is available on the suite. On its own this grades nothing. model: type: - string - 'null' description: >- Resolved judge model. `null` only from deployments that predate resolution. autoRun: type: boolean description: >- The flag that makes grading HAPPEN: grade every run as it completes. Spends per run. threshold: type: number description: Advisory pass threshold, 0–1 (`passed = score >= threshold`). severity: type: string enum: - warn description: >- Presentation severity on the goal-completion slot. Legal only with an advisory role. groundedness: type: object description: >- Stored groundedness slot, when present. Read-only while execution is unwired; PATCH refuses this key. properties: role: type: string enum: - advisory model: type: - string - 'null' threshold: type: - number - 'null' severity: type: string enum: - warn rubric: type: - object - 'null' description: >- The suite's own grading criteria, handed to the judge alongside each case's expected output. `null` when the suite has none. The judge cites `id` in its reasons, so ids are load-bearing; editing them retires the suite's judge calibration. required: - criteria properties: criteria: type: array items: type: object required: - id - label properties: id: type: string label: type: string description: type: string required: type: boolean verdictPolicyVersion: type: integer enum: - 2 description: >- The verdict policy this suite's runs are decided under. `2` grades each case against a `passThreshold` FRACTION over its own `repetitions` and decides validity first (an unmeasurable run is `inconclusive`, not failed). ABSENT means legacy: `minimumAccuracy` (a PERCENT) over `max(case.iterations, minimumIterations)`. The two are not convertible, which is why absence is reported rather than defaulted. verdictPolicyDefaults: type: object description: >- Suite defaults a case inherits under PER-CASE GRADING: the pass threshold, the iteration count, and the evidence requirements that let a run be withheld as `inconclusive`. Present only with `verdictPolicyVersion: 2`, and only as a whole — `repetitions` without `passThreshold` cannot answer what a case is graded against. required: - repetitions - passThreshold properties: repetitions: type: integer minimum: 1 description: Trials per case unless the case overrides `repetitions`. passThreshold: type: number minimum: 0 maximum: 1 description: Fraction of a case's trials that must pass. Never a percent. validity: type: object description: >- When a run counts as measured well enough to decide. DECLARED, not resolved: an omitted field is not "no minimum" but the contract default — `minCompletionRate` 0.8, `maxEvaluatorErrorRate` 0.1, and an omitted `minEligibleTrials` requiring every configured trial attempted plus at least one gradeable trial. The policy a run was actually decided under is on the run's `verdictSummary.validity`. properties: minEligibleTrials: type: integer minimum: 1 minCompletionRate: type: number minimum: 0 maximum: 1 maxEvaluatorErrorRate: type: number minimum: 0 maximum: 1 policy: type: string enum: - legacy - v2 description: >- Which policy decides this suite's runs, said in one word. `legacy` grades by `minimumAccuracy` (a percent) over `max(case.iterations, minimumIterations)`; `v2` grades each case by `passThreshold` (a fraction) over its own `repetitions` and decides validity first. It is also what says which threshold a PATCH may send — the two are refused together. qualityGate: description: >- Live quality-gate policy, or `null` when the suite has none. Absent on older API deployments. A DIFFERENT setting from the pass criteria above: this one compares a run against a baseline run, and a change to it requires an audited reason that appears in the suite's revision history. oneOf: - $ref: '#/components/schemas/SuiteGatePolicyV1' - type: 'null' required: - minimumAccuracy - matchOptions - checks - judge EvalSuiteSchedule: type: object required: - enabled description: >- Recurring-run schedule. A schedule fires exactly ONE run, so an environment-based suite pins the environment it runs against. properties: enabled: type: boolean intervalMinutes: type: - integer - 'null' minimum: 5 maximum: 10080 description: Minutes between runs. 5 minutes to 7 days. environmentId: type: - string - 'null' description: >- The single attached environment scheduled runs use. Null on a legacy (non-environment) suite. state: type: - string - 'null' enum: - active - paused_quota - paused_auth - paused_failures - null description: >- What the schedule is DOING, which `enabled` cannot say. A schedule pauses itself on exhausted quota, on the owner losing access to the suite, and after repeated consecutive failures — all three keep `enabled: true`, so a caller reading only `enabled` reports a healthy automation that has not run in a week. createdBy: type: - string - 'null' description: >- The user id the schedule runs AS. Scheduled runs use this person's access, and the schedule pauses (`paused_auth`) if they lose it. nextDueAt: type: - integer - 'null' description: Epoch ms of the next due firing, or `null` when nothing is due. consecutiveFailures: type: integer description: Consecutive failed firings; resets on the first success. Error: type: object required: - code - message properties: code: type: string description: >- Stable, machine-readable error code. New codes may be added over time; treat unknown codes as non-retryable failures unless the HTTP status says otherwise. enum: - UNAUTHORIZED - FORBIDDEN - NOT_FOUND - CONFLICT - VALIDATION_ERROR - RATE_LIMITED - FEATURE_NOT_SUPPORTED - SERVER_UNREACHABLE - TIMEOUT - OAUTH_REQUIRED - INTERNAL_ERROR message: type: string description: >- Human-readable description. May change between releases — don't match on it. details: type: object description: Optional, unstructured context bag. additionalProperties: true SuiteGatePolicyV1: type: object description: >- Stored quality-gate policy: the conditions a run is failed on for REGRESSING against a baseline run, which is a different question from the pass criteria (`minimumAccuracy` or `verdictPolicyDefaults.passThreshold`). A run can meet its pass criterion and still fail the gate, and the reverse. Every field is optional and an omitted condition is not evaluated at all — it is not a condition set to zero. Numeric fields are FRACTIONS in [0,1] (`maximumPassRateDrop`) or nonnegative milliseconds (`maximumP95LatencyIncreaseMs`); a percent-shaped `3` meaning "3%" is a usage error, not three percentage points. A comparative condition with no resolvable baseline is `non_gateable` rather than passing. properties: baseline: description: >- Where the comparative conditions look for their baseline run. Omitted means there is none, which makes every comparative condition `non_gateable` rather than passing. oneOf: - type: object properties: kind: type: string enum: - run runId: type: string required: - kind - runId additionalProperties: false - type: object properties: kind: type: string enum: - commit_sha commitSha: type: string required: - kind - commitSha additionalProperties: false - type: object description: >- RESERVED. The shape exists so a later capability can enable it without forking the schema; every public write refuses it today, and a historical row that already stores it evaluates as `non_gateable` rather than being silently ignored. properties: kind: type: string enum: - previous_completed required: - kind additionalProperties: false maximumPassRateDrop: type: number description: >- How far the pass rate may fall below the baseline's, as a FRACTION in [0,1]. `0.05` is five percentage points, not five percent of the baseline. noDeterministicRegressions: type: boolean description: >- Fail the run when a scorer that passed deterministically on the baseline does not on this run. maximumP95LatencyIncreaseMs: type: number description: How far p95 latency may rise above the baseline's, in milliseconds. noGatingScoreErrors: type: boolean description: >- Fail the run when any required evaluator errored. An evaluator error is not a server defect — it means the run says nothing about the server — so this is a condition rather than a verdict. additionalProperties: false responses: Unauthorized: description: >- Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key. content: application/json: schema: $ref: '#/components/schemas/Error' examples: badKey: summary: Invalid or revoked key value: code: UNAUTHORIZED message: Invalid API key oauthRequired: summary: Target server needs an OAuth grant value: code: OAUTH_REQUIRED message: Server requires OAuth authorization Forbidden: description: Key is valid but not allowed to do this. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: FORBIDDEN message: You do not have access to this project NotFound: description: Unknown project, server, or resource. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: NOT_FOUND message: Server not found RateLimited: description: >- Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter. headers: Retry-After: description: Seconds to wait before retrying. schema: type: integer content: application/json: schema: $ref: '#/components/schemas/Error' example: code: RATE_LIMITED message: API key rate limit exceeded. Slow down and retry. InternalError: description: Something failed on MCPJam's side. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: INTERNAL_ERROR message: Unexpected internal error ServerUnreachable: description: Could not connect to the target MCP server. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: SERVER_UNREACHABLE message: Failed to connect to server securitySchemes: bearerAuth: type: http scheme: bearer description: >- MCPJam API key (`sk_…`). Create one at [Settings → API keys](https://app.mcpjam.com/settings/api-keys). Guest sessions cannot use the API, and API keys cannot manage other API keys. ````