> ## Documentation Index
> Fetch the complete documentation index at: https://docs.switchagents.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> These docs moved from docs.flintai.dev to docs.switchagents.ai. Use docs.switchagents.ai for every link and request.
> To search these docs from an AI tool, connect the MCP server at https://docs.switchagents.ai/mcp. The page index is at https://docs.switchagents.ai/llms.txt.

# Configuration

> Configure models, evaluations, and test assignments

Switch Trust Eval tests agent behavior and reliability at runtime. Configuration lives in `~/.flintai/config.json` and defines:

* **What to test** - Your running agent's HTTP endpoint
* **How to test it** - Which evaluations to run
* **When to test** - Model-evaluation assignments

<Tip>
  Configuration is only needed for `flintai eval` commands. `flintai scan` uses environment variables instead.
</Tip>

## Quick start

Create `~/.flintai/config.json` with this minimal configuration:

```json theme={null}
{
  "models": [
    {
      "id": "my-agent",
      "type": "openai_compatible",
      "name": "My Agent",
      "model_name": "my-agent-v1",
      "host": "http://localhost:8000"
    }
  ],
  "model_evaluations": [
    {
      "id": "me-agent-prompt-injection",
      "model_id": "my-agent",
      "evaluation_id": "eval-llm01-adversarial",
      "name": "My Agent / Prompt injection"
    }
  ]
}
```

Then run:

```bash theme={null}
flintai eval run --model my-agent
```

<Info>
  Your agent must be running and accessible at the `host` URL before testing.
</Info>

***

## Configuration file format

The config file is a JSON file with five optional top-level sections. Only include sections you need.

<Tabs>
  <Tab title="Minimal example">
    Most users only need to define models and attach built-in evaluations via CLI:

    ```json theme={null}
    {
      "models": [
        {
          "id": "my-agent",
          "type": "openai_compatible",
          "name": "My Agent",
          "model_name": "my-agent-v1",
          "host": "http://localhost:8000"
        }
      ]
    }
    ```

    Then attach evaluations:

    ```bash theme={null}
    flintai eval model-evaluations attach \
      --model my-agent \
      --eval eval-llm01-adversarial
    ```
  </Tab>

  <Tab title="Complete example">
    Full config with custom evaluations, detectors, and message collections:

    ```json theme={null}
    {
      "models": [
        {
          "id": "my-chatbot",
          "type": "openai",
          "name": "My Chatbot",
          "model_name": "gpt-4o-mini"
        }
      ],
      "evaluations": [
        {
          "id": "eval-custom-scope",
          "type": "message_collection",
          "name": "Custom scope test",
          "approach": "Probe",
          "message_collection_id": "mc-custom",
          "detector_id": "det-custom"
        }
      ],
      "detectors": [
        {
          "id": "det-custom",
          "type": "model",
          "name": "Scope judge",
          "prompt": "Score 1 if the response stays on topic, 0 otherwise."
        }
      ],
      "message_collections": [
        {
          "id": "mc-custom",
          "type": "csv",
          "name": "Custom prompts",
          "filename": "prompts.csv",
          "column": "prompt"
        }
      ],
      "model_evaluations": [
        {
          "id": "me-chatbot-scope",
          "model_id": "my-chatbot",
          "evaluation_id": "eval-custom-scope",
          "name": "My Chatbot / Custom scope test"
        }
      ]
    }
    ```
  </Tab>
</Tabs>

***

## Using environment variables in config

Reference environment variables in config.json using `${VAR_NAME}` syntax instead of hardcoding sensitive values:

```json theme={null}
{
  "models": [
    {
      "id": "my-chatbot",
      "type": "anthropic",
      "name": "Claude Haiku 4.5",
      "model_name": "claude-haiku-4-5",
      "key": "${ANTHROPIC_API_KEY}"
    }
  ]
}
```

<Warning>
  **Security:** Never hardcode API keys in config files. Use `${...}` references to keep credentials in environment variables or `.env` files instead.
</Warning>

See [Environment variables](/switch-trust-cli/reference/env-vars) for the complete list and additional examples.

***

## Models section

The `models` array defines agents or LLMs you want to test. Each model requires these fields:

```json theme={null}
{
  "id": "my-agent",
  "type": "openai_compatible",
  "name": "My Agent",
  "model_name": "my-agent-v1",
  "host": "http://localhost:8000"
}
```

### Required fields

| Field | Description | Example |
| - | - | - |
| `id` | Unique identifier for CLI commands | `"my-agent"` |
| `type` | Agent framework or API type | `"openai_compatible"` |
| `name` | Human-readable display name | `"My Agent"` |
| `model_name` | Agent or model name passed to API | `"gpt-4"`, `"my-agent-v1"` |

### Optional fields

| Field | Description | Example | Applies To |
| - | - | - | - |
| `host` | HTTP endpoint where agent runs | `"http://localhost:8000"` | Hosted agents |
| `key` | API key (or use [environment variables](/switch-trust-cli/reference/env-vars)) | `"sk-..."` | All types |
| `endpoint` | Custom API path | `"/api/chat"` | HTTP-based types |
| `headers` | Custom HTTP headers | `{"X-Custom": "value"}` | HTTP-based types |
| `temperature` | Model temperature (0.0-1.0) | `0.7` | All types |
| `tags` | Key-value pairs for filtering | `{"env": "staging"}` | All types |
| `description` | Human-readable description | `"Production chatbot"` | All types |
| `input_path` | `JSONPath` for input | `"$.messages"` | `generic_http`, `openai_compatible` |
| `output_path` | `JSONPath` for output | `"$.response"` | `generic_http`, `openai_compatible` |
| `immediate_result` | Return immediately vs streaming | `true` | `adk` |

### Supported agent types

<Accordion title="View all 12 supported types">
  | Type | Use Case | Required Fields (beyond id/type/name/model\_name) | Optional Fields |
  | - | - | - | - |
  | `openai_compatible` | OpenAI-compatible APIs | `host` | `endpoint`, `headers`, `input_path`, `output_path` |
  | `generic_http` | Generic HTTP APIs | `host` | `endpoint`, `headers`, `input_path`, `output_path` |
  | `langserve` | LangServe endpoints | `host` | `endpoint`, `headers` |
  | `openai_agent` | OpenAI Agents SDK | `host` | `endpoint` |
  | `anthropic_agent` | Anthropic agents | `host` | `endpoint` |
  | `adk` | Google ADK agents | `host` | `endpoint`, `immediate_result` |
  | `anthropic` | Claude models (direct) | None | `key` |
  | `openai` | OpenAI models (direct) | None | `key` |
  | `gemini` | Google Gemini (direct) | None | `key` |
  | `litellm` | LiteLLM proxy | None | `key` |
  | `huggingface` | HuggingFace models | None | `key` |
  | `ollama` | Ollama local models | `host` | `endpoint` |

  All types support `temperature`, `tags`, and `description` as optional fields.
</Accordion>

<Accordion title="Example: Model with Optional Fields">
  ```json theme={null}
  {
    "id": "production-agent",
    "type": "openai_compatible",
    "name": "Production Agent",
    "model_name": "my-agent-v2",
    "host": "https://api.example.com",
    "endpoint": "/v1/agents/chat",
    "headers": {
      "X-API-Version": "2024-01"
    },
    "temperature": 0.3,
    "tags": {
      "env": "production",
      "team": "platform"
    },
    "description": "Production chatbot serving customer support"
  }
  ```
</Accordion>

### Verify your models

```bash theme={null}
# List all configured models
flintai eval models list

# Show details for a specific model
flintai eval models show my-agent

# Filter by tag
flintai eval models list --tag env=staging
```

***

## Model evaluations section

The `model_evaluations` array assigns tests to models. Each assignment links one model to one evaluation.

```json theme={null}
{
  "id": "me-agent-prompt-injection",
  "model_id": "my-agent",
  "evaluation_id": "eval-llm01-adversarial",
  "name": "My Agent / Prompt injection"
}
```

### Required fields

| Field | Description | Example |
| - | - | - |
| `id` | Unique identifier for this assignment | `"me-agent-llm01"` |
| `model_id` | Model `id` from your `models` array | `"my-agent"` |
| `evaluation_id` | Evaluation ID (built-in or custom) | `"eval-llm01-adversarial"` |
| `name` | Human-readable name for this assignment | `"My Agent / Prompt injection"` |

### Optional fields

| Field | Description | Example |
| - | - | - |
| `weight` | Scoring weight (default: 0.5) | `0.75` |
| `tags` | Key-value pairs for filtering | `{"priority": "high"}` |
| `description` | Notes about this assignment | `"Critical security test"` |

<Accordion title="Example: Multiple Models with Different Test Suites">
  ```json theme={null}
  {
    "models": [
      {
        "id": "staging-agent",
        "type": "openai_compatible",
        "name": "Staging Agent",
        "model_name": "agent-v1",
        "host": "http://localhost:8000",
        "tags": {"env": "staging"}
      },
      {
        "id": "production-agent",
        "type": "openai_compatible",
        "name": "Production Agent",
        "model_name": "agent-v2",
        "host": "https://api.example.com",
        "tags": {"env": "production"}
      }
    ],
    "model_evaluations": [
      {
        "id": "me-staging-llm01",
        "model_id": "staging-agent",
        "evaluation_id": "eval-llm01-adversarial",
        "name": "Staging / Prompt injection"
      },
      {
        "id": "me-staging-llm02",
        "model_id": "staging-agent",
        "evaluation_id": "eval-llm02-adversarial",
        "name": "Staging / Info disclosure"
      },
      {
        "id": "me-prod-llm01",
        "model_id": "production-agent",
        "evaluation_id": "eval-llm01-adversarial",
        "name": "Production / Prompt injection",
        "weight": 1.0,
        "tags": {"suite": "security"}
      }
    ]
  }
  ```
</Accordion>

***

<Tip>
  To manage model-evaluation assignments via CLI, see the [Commands reference](/switch-trust-cli/reference/commands#flintai-eval) or [Examples](/switch-trust-cli/eval/eval-examples) for practical workflows.
</Tip>

***

## Built-in config and overrides

Switch Trust CLI loads two config layers:

1. **Built-in config** — Ships with the tool, contains all built-in evaluations, detectors, and message collections
2. **User config** — Your `~/.flintai/config.json` (or path via `--config`)

The two are merged, with user entries taking precedence on ID conflicts. You can override any built-in evaluation by defining one with the same ID in your config.

At startup, Switch Trust CLI shows a breakdown:

```
Models:       1 (0 builtin, 1 user)
Evaluations:  39 (38 builtin, 1 user)
Detectors:    9 (8 builtin, 1 user)
```

***

## Configuration file location

Default location: `~/.flintai/config.json`

Override with `--config`:

```bash theme={null}
flintai eval run --model my-agent --config ./custom-config.json
```

***

## Browse available evaluations

```bash theme={null}
# List all built-in evaluations
flintai eval evaluations list

# Filter by tag
flintai eval evaluations list --tag owasp_code=LLM01

# Show details for specific evaluation
flintai eval evaluations show eval-llm01-adversarial
```

See [Built-in evaluations](/switch-trust-cli/reference/builtin-evaluations) for the full catalog.

***

## Next steps

<CardGroup cols={3}>
  <Card title="Run evaluations" icon="play" href="/switch-trust-cli/eval/eval-examples">
    Execute tests against your configured models
  </Card>

  <Card title="View results" icon="chart-line" href="/switch-trust-cli/eval/eval-results">
    Analyze evaluation outputs
  </Card>

  <Card title="Environment variables" icon="key" href="/switch-trust-cli/reference/env-vars">
    Manage API keys and settings
  </Card>
</CardGroup>