CLI Reference

MCPJam Inspector includes a dedicated Command Line Interface (CLI) for programmatic MCP testing, designed for integration into CI/CD pipelines and local development workflows.

Installation

To use the CLI, install it globally via npm:

npm install -g @mcpjam/cli

Usage

The primary command for the CLI is evals run, which executes a suite of tests against one or more MCP servers.

evals run

mcpjam evals run --tests <tests.json> --environment <env.json> --api-key <your_key>

Options

Flag	Alias	Description
--tests <file>	-t	(Required) Path to the JSON file containing your test definitions.
--environment <file>	-e	(Required) Path to the JSON file defining your MCP servers and provider API keys.
--api-key <key>	-a	(Required) Your personal access key for the MCPJam platform.

Configuration Files

The CLI uses two JSON files to configure a test run.

Tests File (--tests)

This file defines the test cases to be executed. Each test includes a prompt, the expected tools to be called, and the model to use.

Example (test-servers.json):

{
  "tests": [
    {
        "title": "Test workspaces",
        "query": "Can you list my asana workspace?",
        "runs": 1,
        "model": "anthropic/claude-3.5-sonnet",
        "expectedToolCalls": [
            "asana_list_workspaces"
        ],
        "judgeRequirement": "The workspace was successfully returned",
        "advancedConfig": {
            "instructions": "You are a helpful weather assistant",
            "temperature": 0.1,
            "maxSteps": 5,
            "toolChoice": "auto"
        }
    }
]
}

Environment File (--environment)

This file defines the MCP servers to be tested and the API keys for the different LLM providers.

Example (mcp-environment.json):

{
  "servers": {
    "asana": {
      "url": "https://mcp.asana.com/sse",
      "requestInit": {
          "headers": {
              "Authorization": "Bearer <YOUR_ASANA_TOKEN>"
          }
      }
    }
  }
}

Output

The CLI provides a summary of the test run, indicating which tests passed or failed and the total duration.

MCPJAM Evals v1.0.0

Running 3 tests against weather-server...

✅ Weather tool functionality
   Called tools: get_weather
   Duration: 1.2s

❌ Error handling test
   Called tools: get_weather, validate_location
   Missing: []
   Unexpected: validate_location
   Duration: 0.8s

Results: 2 passed, 1 failed (2.0s total)