> ## Documentation Index
> Fetch the complete documentation index at: https://docs.switchagents.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> These docs moved from docs.flintai.dev to docs.switchagents.ai. Use docs.switchagents.ai for every link and request.
> To search these docs from an AI tool, connect the MCP server at https://docs.switchagents.ai/mcp. The page index is at https://docs.switchagents.ai/llms.txt.

# Eval your agent

> Get proof your agent is production-ready

Test factual accuracy, instruction adherence, prompt injection, jailbreaks, and [more](/switch-trust-cli/reference/builtin-evaluations). Tests are framework-agnostic and provide a 0.0-1.0 score proving agent reliability.

<Card title="Switch Trust CLI on GitHub" icon="github" href="https://github.com/sandbox-quantum/flintai-cli">
  Source code, example agents, and issue tracking
</Card>

<Tip>
  Install our MCP server in Claude Code or your AI code assistant, then ask: **"Help me set up Switch Trust Eval"** to get live guidance, troubleshoot issues, and work through these steps together. [Learn how →](/use-these-docs)
</Tip>

## Evaluate your agent at runtime

<Note>
  **Before you start:** Your agent must be running and accessible over HTTP. Evaluation works by sending prompts to a live endpoint and scoring the responses.
</Note>

<Steps>
  <Step title="Verify flintai-cli is installed">
    ```bash theme={null}
    flintai --version
    ```

    <Accordion title="Need setup?">
      If not installed, using a virtual environment is recommended to avoid dependency conflicts:

      ```bash theme={null}
      python3 -m venv .venv
      source .venv/bin/activate
      ```

      Install Switch Trust CLI:

      ```bash theme={null}
      pip install flintai-cli
      flintai init
      ```

      [Full installation guide →](/switch-trust-cli#try-it-now)
    </Accordion>

    <Note>
      **Bring your own LLM key.** Evaluation uses an LLM as judge, so it needs an API key for your own LLM provider (Gemini, OpenAI, Anthropic, or LiteLLM). This is your provider key for the local CLI, not a Switch Trust API token. `flintai init` prompts for it and stores it locally in `~/.flintai/.env`.
    </Note>

    <Accordion title="Where to get an LLM provider key">
      * **Google Gemini**: [aistudio.google.com/apikey](https://aistudio.google.com/apikey) (free tier available)
      * **OpenAI**: [platform.openai.com/api-keys](https://platform.openai.com/api-keys)
      * **Anthropic**: [console.anthropic.com/settings/keys](https://console.anthropic.com/settings/keys)
    </Accordion>
  </Step>

  <Step title="Start your agent">
    Switch Trust Eval sends prompts to your agent over HTTP, so it must be running and reachable before you evaluate. Start your agent and note the host and port it listens on — you'll add these to your config next.

    <Accordion title="Try it with a shipped example agent">
      No agent of your own yet? Clone [flintai-cli](https://github.com/sandbox-quantum/flintai-cli) and serve the bookstore example (OpenAI Agents SDK) on port 8010 from the repo root:

      ```bash theme={null}
      uvx --with openai-agents,fastapi --from uvicorn uvicorn examples.bookstore_agent.agent:app --port 8010 --host 0.0.0.0
      ```

      <Note>
        This command uses `uvx`, which ships with [uv](https://docs.astral.sh/uv/getting-started/installation/). Install uv first if you don't have it.
      </Note>

      The agent keeps running in this terminal and is now reachable at `http://localhost:8010`. In a separate terminal, run the bundled evaluations. The repo ships `examples/config.json` with both example agents pre-configured, so you can skip **Add your agent** and **Attach evaluations** and run directly:

      ```bash theme={null}
      flintai eval run --model model-bookstore-agent --config examples/config.json
      ```

      This config attaches OWASP LLM01–LLM09, PII, and secrets evaluations. The `weather_agent` (Google ADK) example is included too.
    </Accordion>
  </Step>

  <Step title="Add your agent">
    Create or update your agent config file with connection details:

    ```json theme={null}
    {
      "models": [
        {
          "id": "my-agent",
          "type": "adk",
          "name": "My Agent",
          "host": "http://localhost:8000"
        }
      ]
    }
    ```

    <Note>
      **Important:** The `host` field must match where your agent is actually running.
    </Note>

    <Accordion title="How do I edit my config file?">
      The config file is stored in `~/.flintai/config.json` (where `~` means your home directory).

      <Tip>
        Folders starting with a dot are hidden from Finder and File Explorer. Use the commands below to create and open the file automatically.
      </Tip>

      <Tabs>
        <Tab title="macOS / Linux">
          These commands create the `.flintai` directory if needed, then open the config file in TextEdit:

          ```bash theme={null}
          mkdir -p ~/.flintai
          open -e ~/.flintai/config.json
          ```

          Add your agent's connection details and save (Cmd+S or File → Save).
        </Tab>

        <Tab title="Windows PowerShell">
          These commands create the `.flintai` directory if needed, then open the config file in Notepad:

          ```powershell theme={null}
          New-Item -ItemType Directory -Force "$HOME\.flintai" | Out-Null
          notepad "$HOME\.flintai\config.json"
          ```

          Add your agent's connection details and save (Ctrl+S or File → Save).
        </Tab>
      </Tabs>
    </Accordion>

    <Accordion title="What do these config fields mean?">
      * `id` - Unique ID for this model or agent. You'll use it in commands like `--model my-agent`.
      * `type` - Your agent's framework (expand supported types below).
      * `name` - The label that will identify this agent in results and logs.
      * `host` - Base URL for the target endpoint, if this type connects over HTTP.

      <Note>
        Start your agent and confirm it answers at the `host` URL before you run an evaluation.
      </Note>
    </Accordion>

    <Accordion title="Supported agent types">
      * **adk** - Google ADK agents
      * **openai\_agent** - OpenAI Agents SDK
      * **langchain** - LangChain agents
      * **crewai** - CrewAI agents

      See [Configuration](/switch-trust-cli/eval/eval-configuration) for all types and options.
    </Accordion>
  </Step>

  <Step title="Attach evaluations">
    Browse [built-in evaluations](/switch-trust-cli/reference/builtin-evaluations) to see available tests, then attach them to your agent:

    <Note>
      No evaluations run by default. You must attach at least one evaluation before running `flintai eval run`.
    </Note>

    ```bash theme={null}
    flintai eval model-evaluations attach \
      --model my-agent \
      --eval eval-llm09-fixed

    flintai eval model-evaluations attach \
      --model my-agent \
      --eval eval-llm01-fixed
    ```

    <Accordion title="How do I attach multiple evaluations at once?">
      Use `--eval-tag` to batch-attach evaluations by tag:

      ```bash theme={null}
      flintai eval model-evaluations attach --model my-agent --eval-tag owasp_code=LLM01
      ```

      This attaches all evaluations tagged with `owasp_code=LLM01` (prompt injection tests) in a single command. See [built-in evaluations](/switch-trust-cli/reference/builtin-evaluations) for all available tests and tags.
    </Accordion>
  </Step>

  <Step title="Run evaluation">
    Execute all attached tests:

    ```bash theme={null}
    flintai eval run --model my-agent
    ```

    `flintai eval` sends test prompts to your agent, judges the responses using LLM-as-judge, and scores reliability on a 0.0-1.0 scale.

    Evaluations can take several minutes depending on the number of tests. Progress updates appear in the CLI, and a summary displays when complete. Results are saved to `eval_<timestamp>.json`, or `eval_<timestamp>.sarif` if you pass `--format sarif`.
  </Step>
</Steps>

<Tip>
  **Integrate with CI/CD.** Save eval results as build artifacts to prove agent reliability before deployment. [See CI/CD integration guide →](/switch-trust-cli/guides/ci-cd-integration)
</Tip>

## Ship with confidence

**What the score means:**

* **0.8+** - Production-ready
* **0.6-0.8** - Needs improvement
* **\<0.6** - Not ready for production

**Next steps:**

<CardGroup cols={3}>
  <Card title="Interpret your results" icon="chart-line" href="/switch-trust-cli/eval/eval-results">
    Understand score breakdowns and track improvement over time
  </Card>

  <Card title="How evaluation works" icon="diagram-project" href="/switch-trust-cli/eval/how-evaluation-works">
    Learn the LLM-as-judge methodology and scoring calculation
  </Card>

  <Card title="Scan agent code" icon="magnifying-glass" href="/switch-trust-cli/scan/getting-started">
    Find agent code issues before deployment with Switch Trust Scan
  </Card>
</CardGroup>