> ## Documentation Index
> Fetch the complete documentation index at: https://docs.switchagents.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> These docs moved from docs.flintai.dev to docs.switchagents.ai. Use docs.switchagents.ai for every link and request.
> To search these docs from an AI tool, connect the MCP server at https://docs.switchagents.ai/mcp. The page index is at https://docs.switchagents.ai/llms.txt.
> The source code for these products is public on GitHub. Switch Rooms: https://github.com/sandbox-quantum/switch. Switch Trust CLI: https://github.com/sandbox-quantum/switch-trust-cli. Switch Trust CodeScan action: https://github.com/sandbox-quantum/switch-trust-codescan-action. Switch Trust MCP server: https://github.com/sandbox-quantum/switch-trust-mcp. Switch Trust Python SDK: https://github.com/sandbox-quantum/switch-trust-sdk-py. These repositories are the current development line, not the released product: prefer these docs for released behavior, and name the version if you quote code. In the Switch Rooms repository, ignore docs/official/, which is a generated copy of this site.

# Configuration

> Configure models, evaluations, and test assignments

Switch Trust Eval tests agent behavior and reliability at runtime. Configuration lives in `~/.switch-trust/config.json` and defines:

* **What to test** - Your running agent's HTTP endpoint
* **How to test it** - Which evaluations to run
* **When to test** - Model-evaluation assignments

> [!TIP]
>
> Configuration is only needed for `switch-trust eval` commands. `switch-trust scan` uses environment variables instead.

## Quick start

Create `~/.switch-trust/config.json` with this minimal configuration:

```json theme={null}
{
  "models": [
    {
      "id": "my-agent",
      "type": "openai_compatible",
      "name": "My Agent",
      "model_name": "my-agent-v1",
      "host": "http://localhost:8000"
    }
  ],
  "model_evaluations": [
    {
      "id": "me-agent-prompt-injection",
      "model_id": "my-agent",
      "evaluation_id": "eval-llm01-adversarial",
      "name": "My Agent / Prompt injection"
    }
  ]
}
```

Then run:

```bash theme={null}
switch-trust eval run --model my-agent
```

> [!NOTE]
>
> Your agent must be running and accessible at the `host` URL before testing.

***

## Configuration file format

The config file is a JSON file with five optional top-level sections. Only include sections you need.

> [tab] Minimal example
>
> Most users only need to define models and attach built-in evaluations via CLI:
>
> ```json theme={null}
> {
>   "models": [
>     {
>       "id": "my-agent",
>       "type": "openai_compatible",
>       "name": "My Agent",
>       "model_name": "my-agent-v1",
>       "host": "http://localhost:8000"
>     }
>   ]
> }
> ```
>
> Then attach evaluations:
>
> ```bash theme={null}
> switch-trust eval model-evaluations attach \
>   --model my-agent \
>   --eval eval-llm01-adversarial
> ```

> [tab] Complete example
>
> Full config with custom evaluations, detectors, and message collections:
>
> ```json theme={null}
> {
>   "models": [
>     {
>       "id": "my-chatbot",
>       "type": "openai",
>       "name": "My Chatbot",
>       "model_name": "gpt-4o-mini"
>     }
>   ],
>   "evaluations": [
>     {
>       "id": "eval-custom-scope",
>       "type": "message_collection",
>       "name": "Custom scope test",
>       "approach": "Probe",
>       "message_collection_id": "mc-custom",
>       "detector_id": "det-custom"
>     }
>   ],
>   "detectors": [
>     {
>       "id": "det-custom",
>       "type": "model",
>       "name": "Scope judge",
>       "prompt": "Score 1 if the response stays on topic, 0 otherwise."
>     }
>   ],
>   "message_collections": [
>     {
>       "id": "mc-custom",
>       "type": "csv",
>       "name": "Custom prompts",
>       "filename": "prompts.csv",
>       "column": "prompt"
>     }
>   ],
>   "model_evaluations": [
>     {
>       "id": "me-chatbot-scope",
>       "model_id": "my-chatbot",
>       "evaluation_id": "eval-custom-scope",
>       "name": "My Chatbot / Custom scope test"
>     }
>   ]
> }
> ```

***

## Using environment variables in config

Reference environment variables in config.json using `${VAR_NAME}` syntax instead of hardcoding sensitive values:

```json theme={null}
{
  "models": [
    {
      "id": "my-chatbot",
      "type": "anthropic",
      "name": "Claude Haiku 4.5",
      "model_name": "claude-haiku-4-5",
      "key": "${ANTHROPIC_API_KEY}"
    }
  ]
}
```

> [!WARNING]
>
> **Security:** Never hardcode API keys in config files. Use `${...}` references to keep credentials in environment variables or `.env` files instead.

See [Environment variables](https://docs.switchagents.ai/switch-trust-cli/reference/env-vars.md) for the complete list and additional examples.

***

## Models section

The `models` array defines agents or LLMs you want to test. Each model requires these fields:

```json theme={null}
{
  "id": "my-agent",
  "type": "openai_compatible",
  "name": "My Agent",
  "model_name": "my-agent-v1",
  "host": "http://localhost:8000"
}
```

### Required fields

| Field | Description | Example |
| - | - | - |
| `id` | Unique identifier for CLI commands | `"my-agent"` |
| `type` | Agent framework or API type | `"openai_compatible"` |
| `name` | Human-readable display name | `"My Agent"` |
| `model_name` | Agent or model name passed to API | `"gpt-4"`, `"my-agent-v1"` |

### Optional fields

| Field | Description | Example | Applies To |
| - | - | - | - |
| `host` | HTTP endpoint where agent runs | `"http://localhost:8000"` | Hosted agents |
| `key` | API key (or use [environment variables](https://docs.switchagents.ai/switch-trust-cli/reference/env-vars.md)) | `"sk-..."` | All types |
| `endpoint` | Custom API path | `"/api/chat"` | HTTP-based types |
| `headers` | Custom HTTP headers | `{"X-Custom": "value"}` | HTTP-based types |
| `temperature` | Model temperature (0.0-1.0) | `0.7` | All types |
| `tags` | Key-value pairs for filtering | `{"env": "staging"}` | All types |
| `description` | Human-readable description | `"Production chatbot"` | All types |
| `input_path` | `JSONPath` for input | `"$.messages"` | `generic_http`, `openai_compatible` |
| `output_path` | `JSONPath` for output | `"$.response"` | `generic_http`, `openai_compatible` |
| `immediate_result` | Return immediately vs streaming | `true` | `adk` |

### Supported agent types

> [accordion] View all 12 supported types
>
> | Type | Use Case | Required Fields (beyond id/type/name/model\_name) | Optional Fields |
> | - | - | - | - |
> | `openai_compatible` | OpenAI-compatible APIs | `host` | `endpoint`, `headers`, `input_path`, `output_path` |
> | `generic_http` | Generic HTTP APIs | `host` | `endpoint`, `headers`, `input_path`, `output_path` |
> | `langserve` | LangServe endpoints | `host` | `endpoint`, `headers` |
> | `openai_agent` | OpenAI Agents SDK | `host` | `endpoint` |
> | `anthropic_agent` | Anthropic agents | `host` | `endpoint` |
> | `adk` | Google ADK agents | `host` | `endpoint`, `immediate_result` |
> | `anthropic` | Claude models (direct) | None | `key` |
> | `openai` | OpenAI models (direct) | None | `key` |
> | `gemini` | Google Gemini (direct) | None | `key` |
> | `litellm` | LiteLLM proxy | None | `key` |
> | `huggingface` | HuggingFace models | None | `key` |
> | `ollama` | Ollama local models | `host` | `endpoint` |
>
> All types support `temperature`, `tags`, and `description` as optional fields.

> [accordion] Example: Model with Optional Fields
>
> ```json theme={null}
> {
>   "id": "production-agent",
>   "type": "openai_compatible",
>   "name": "Production Agent",
>   "model_name": "my-agent-v2",
>   "host": "https://api.example.com",
>   "endpoint": "/v1/agents/chat",
>   "headers": {
>     "X-API-Version": "2024-01"
>   },
>   "temperature": 0.3,
>   "tags": {
>     "env": "production",
>     "team": "platform"
>   },
>   "description": "Production chatbot serving customer support"
> }
> ```

### Verify your models

```bash theme={null}
# List all configured models
switch-trust eval models list

# Show details for a specific model
switch-trust eval models show my-agent

# Filter by tag
switch-trust eval models list --tag env=staging
```

***

## Model evaluations section

The `model_evaluations` array assigns tests to models. Each assignment links one model to one evaluation.

```json theme={null}
{
  "id": "me-agent-prompt-injection",
  "model_id": "my-agent",
  "evaluation_id": "eval-llm01-adversarial",
  "name": "My Agent / Prompt injection"
}
```

### Required fields

| Field | Description | Example |
| - | - | - |
| `id` | Unique identifier for this assignment | `"me-agent-llm01"` |
| `model_id` | Model `id` from your `models` array | `"my-agent"` |
| `evaluation_id` | Evaluation ID (built-in or custom) | `"eval-llm01-adversarial"` |
| `name` | Human-readable name for this assignment | `"My Agent / Prompt injection"` |

### Optional fields

| Field | Description | Example |
| - | - | - |
| `weight` | Scoring weight (default: 0.5) | `0.75` |
| `tags` | Key-value pairs for filtering | `{"priority": "high"}` |
| `description` | Notes about this assignment | `"Critical security test"` |

> [accordion] Example: Multiple Models with Different Test Suites
>
> ```json theme={null}
> {
>   "models": [
>     {
>       "id": "staging-agent",
>       "type": "openai_compatible",
>       "name": "Staging Agent",
>       "model_name": "agent-v1",
>       "host": "http://localhost:8000",
>       "tags": {"env": "staging"}
>     },
>     {
>       "id": "production-agent",
>       "type": "openai_compatible",
>       "name": "Production Agent",
>       "model_name": "agent-v2",
>       "host": "https://api.example.com",
>       "tags": {"env": "production"}
>     }
>   ],
>   "model_evaluations": [
>     {
>       "id": "me-staging-llm01",
>       "model_id": "staging-agent",
>       "evaluation_id": "eval-llm01-adversarial",
>       "name": "Staging / Prompt injection"
>     },
>     {
>       "id": "me-staging-llm02",
>       "model_id": "staging-agent",
>       "evaluation_id": "eval-llm02-adversarial",
>       "name": "Staging / Info disclosure"
>     },
>     {
>       "id": "me-prod-llm01",
>       "model_id": "production-agent",
>       "evaluation_id": "eval-llm01-adversarial",
>       "name": "Production / Prompt injection",
>       "weight": 1.0,
>       "tags": {"suite": "security"}
>     }
>   ]
> }
> ```

***

> [!TIP]
>
> To manage model-evaluation assignments via CLI, see the [Commands reference](https://docs.switchagents.ai/switch-trust-cli/reference/commands.md#switch-trust-eval) or [Examples](https://docs.switchagents.ai/switch-trust-cli/eval/eval-examples.md) for practical workflows.

***

## Built-in config and overrides

Switch Trust CLI loads two config layers:

1. **Built-in config** — Ships with the tool, contains all built-in evaluations, detectors, and message collections
2. **User config** — Your `~/.switch-trust/config.json` (or path via `--config`)

The two are merged, with user entries taking precedence on ID conflicts. You can override any built-in evaluation by defining one with the same ID in your config.

At startup, Switch Trust CLI shows a breakdown:

```
Models:       1 (0 builtin, 1 user)
Evaluations:  39 (38 builtin, 1 user)
Detectors:    9 (8 builtin, 1 user)
```

***

## Configuration file location

Default location: `~/.switch-trust/config.json`

Override with `--config`:

```bash theme={null}
switch-trust eval run --model my-agent --config ./custom-config.json
```

***

## Browse available evaluations

```bash theme={null}
# List all built-in evaluations
switch-trust eval evaluations list

# Filter by tag
switch-trust eval evaluations list --tag owasp_code=LLM01

# Show details for specific evaluation
switch-trust eval evaluations show eval-llm01-adversarial
```

See [Built-in evaluations](https://docs.switchagents.ai/switch-trust-cli/reference/builtin-evaluations.md) for the full catalog.

***

## Next steps

> [card] [Run evaluations](https://docs.switchagents.ai/switch-trust-cli/eval/eval-examples.md)
>
> Execute tests against your configured models

> [card] [View results](https://docs.switchagents.ai/switch-trust-cli/eval/eval-results.md)
>
> Analyze evaluation outputs

> [card] [Environment variables](https://docs.switchagents.ai/switch-trust-cli/reference/env-vars.md)
>
> Manage API keys and settings
