Switch Trust CLI changelog

• Standard exit codes - switch-trust scan and switch-trust eval run return a distinct exit code for each outcome: a clean run, a tripped gate, an evaluation with no result, a problem to fix, or a temporary failure worth retrying. See Exit codes
• Fail a build on findings or scores - switch-trust scan --fail-on <severity> and switch-trust eval run --fail-under <score> exit with code 5 when a finding or score crosses the threshold you set. The results file is still written first. See Fail the build on findings or scores

• Clearer errors - An error shows in a panel with its exit code and what to do next
• Fewer false PII detections - The evaluation PII detector flags fewer things that aren’t personal data
• A warning when no gate is set - In CI, a run without --fail-on or --fail-under warns that it always exits 0
• google-adk dependency - Now pinned to 1.38.0, down from 2.2.0. If your project pins a newer google-adk, check for a conflict

⚠️ Action required

• Incomplete runs now exit non-zero - Before 1.8.0, these runs exited 0 without producing a result, so a pipeline that treated 0 as success, or worked around it by reading the results file, behaves differently now. Each now exits with a code, and a message that says what to fix:
  • A scanner tool is missing, such as OpenGrep, Bandit, detect-secrets, or pip-audit: exits 26. Install the tool.
  • A configured scanner model has no API key, or its key is rejected mid-scan: exits 20, or another configuration or retry code. Set or replace the key.
  • An evaluation run scored nothing, or stopped before every evaluation ran: exits 6. The message names the cause, usually the agent endpoint or its credentials.
  • The scan path doesn’t exist: exits 22 instead of 1. Fix the path.
  Where a scan ran, it still writes its results first. Findings and scores still never fail a build unless you add --fail-on or --fail-under: a scan that ran fully exits 0, whatever it found. There’s no flag to restore the old behavior. To keep it, pin switch-trust-cli==1.7.2. See Exit codes

• Corrected project links - The package’s homepage and issue-tracker links point to the switch-trust-cli repository, and the README uses the current Switch Trust branding

• Flint AI CLI is now Switch Trust CLI - Breaking: The package is now switch-trust-cli (pip install switch-trust-cli) and the command is switch-trust, so flintai scan becomes switch-trust scan and flintai eval becomes switch-trust eval. The config directory moves from ~/.flintai to ~/.switch-trust — re-run switch-trust init or move the directory, as there is no automatic fallback. The FLINTAI_TELEMETRY_CONSENT and FLINTAI_CLIENT_ID environment variables become SWITCH_TRUST_TELEMETRY_CONSENT and SWITCH_TRUST_CLIENT_ID; the old names still work for now and print a one-time deprecation notice. See environment variables
• Steadier evaluation against live agents - Evaluation limits how many requests it sends to a single agent endpoint at once and spreads out the opening burst, so a run stays within the agent’s rate limit. This covers the HTTP-based model types — ADK, OpenAI agent, Anthropic agent, generic HTTP, LangServe, and Vertex AI Agent Runtime; SDK-based models are unaffected
• Smarter retries - Evaluation honors a server’s Retry-After response after a rate limit, and no longer retries authentication or not-found errors (401, 403, 404), so a bad key or wrong endpoint fails fast instead of looping

• Evaluation says why a run failed - A failed run now reports the specific reason — no prompts to run, every prompt failed, or too few prompts succeeded to score — instead of a generic empty result
• Clearer errors from agent endpoints - When an agent’s HTTP endpoint returns an error, the CLI surfaces the useful detail from the response without logging your content

• Scan findings point to the right file in SARIF - A scan finding now uses the actual evidence file path as its location in SARIF output, instead of a free-text label that varied between runs for the same issue. Component names are kept under properties.affectedComponents. See scan results
• More accurate finding categories - A finding’s category is derived from its subcategory and mapped to the canonical taxonomy, so a correct finding is no longer mis-scored when the model names the wrong category; an unrecognized category is flagged rather than dropped

• Evaluate agents on Google Vertex AI Agent Runtime - Point flintai eval at an agent deployed to Vertex AI Agent Runtime (Reasoning Engine), authenticating with either a service-account key or an OAuth access token
• Custom headers for ADK agents - The ADK model type accepts custom HTTP headers, so you can reach a deployment behind an app-level token such as an Authorization bearer

• An unreliable run is flagged, not scored - When fewer than 90% of an evaluation’s prompts succeed, the run is reported as an error instead of returning a score computed from a fraction of the prompts
• The judge resists prompt injection - The response under evaluation is fenced off and the LLM judge is instructed to treat it strictly as data, so content in a model’s response can’t steer the judge
• Deeper scan reasoning - The scanner reasons for longer before concluding, improving coverage on larger agents
• Faster garak evaluations - Garak detector datasets load once per process instead of reloading for every job

• Dependency update - The bundled transformers library, used by the optional full extra, was updated to a newer release

• Reasoning-based scanning - flintai scan now investigates an agent the way an application security researcher would — following untrusted input into the agent, tool output fed back to the model, unbounded loops, and missing authorization or guardrails — instead of running a fixed checklist, which broadens what a scan finds. See scan results
• Reliable structured output from eval models - Evaluation asks each provider for structured JSON directly (OpenAI, Anthropic, Gemini, Ollama, LiteLLM, and vLLM), so judge and detector results parse reliably, falling back to text only when a provider can’t
• Judges explain their score - An LLM-judge detector returns a reason alongside its score, so you can see why a response was marked the way it was
• Errored prompts no longer drag down a score - A prompt that errors during evaluation is excluded from the average instead of counting as a zero
• Adversarial content isn’t blocked mid-eval - The Gemini generator and judge no longer apply safety filters to red-team content, so a blocked judge no longer surfaces as a spurious error. This affects only the generator and judge, not the model under test
• Updated default models for new setups - flintai init writes current model defaults into a new config; existing configs keep the models they already pin

• Clearer failures when a key or model is wrong - Scan and evaluation classify a failure as authentication, model, provider, or input and give a precise message; a missing API key skips the LLM step with a warning instead of crashing

• Per-project configuration - A .env in your working directory now configures that project, layered over the global ~/.flintai/.env. Anything already set in your shell or CI still wins. See environment variables

• Quicker to start - flintai --version, flintai --help, and a mistyped argument return without loading the evaluation and telemetry stack, and flintai scan loads the scanner only once a scan runs

• Scans complete on GPT-5 reasoning models - Tool calls reach the interface that supports them, and a parameter the model declines is dropped rather than ending the scan

• Leaner default install - Breaking: The standard install is roughly 2 GB smaller and no longer pulls CUDA on Linux. Scanning and the built-in OWASP LLM evaluations work as before. Garak evaluations, toxicity detection, and locally run HuggingFace models now need the optional full extra. See installation

• Pseudonymous usage analytics - The CLI asks on first run whether to share pseudonymous usage data, and pressing Enter turns it on. You can change your answer at any time, CI runs never send, and code, prompts, and keys are never collected. See data privacy
• Incomplete scans now say so - When an analysis tool is unavailable, flintai scan warns that the findings are partial and names what’s missing, instead of quietly returning a shorter list. See scan results
• --log on flintai scan - Set the log file path, matching the flag already available on flintai eval

• Default scan model - New setups scan with gemini-3.6-flash. Existing setups keep the model they already pin. See environment variables
• Eval results name the schema version in snake_case - Breaking: the schemaVersion field in eval JSON output is now schema_version, and the format is version 2.0. Update automation that reads the old key. See eval results
• Scan JSON drops two empty fields - Breaking: agents_found and agent_profiles are gone from flintai scan JSON output. Both were always empty. See scan results
• Predictable dependency resolution - The OpenTelemetry and protobuf version ranges are pinned so pip install resolves without backtracking

• Scanning a single file returns findings - flintai scan path/to/agent.py previously completed and reported nothing. A single file is now analyzed the same way it would be inside a directory. See the command reference

• Dependency hardening - Cryptography and networking libraries updated to current maintained releases

• Scan finding impact text - The impact field on a scan finding now carries the description from the issue taxonomy instead of a generated Exploitable via ... string. This text appears in both JSON and SARIF output, so update any automation that matches on the old wording
• Safety refusals during eval - A model refusal for a disallowed prompt now reports as blocked for safety rather than as a model error, so an eval run distinguishes a refusal from a failure

• Eval no longer stops on a malformed provider error - An error response with an unexpected body is handled instead of ending the run
• Corrected documentation links - Links in the project README now point to migrated documentation paths

• --format {json,sarif} - Choose the output format for flintai scan and flintai eval run, with -f as the short form. json stays the default, so existing commands behave as before
• SARIF 2.1.0 output - Feed scan and eval results directly into GitHub code scanning and other SARIF-aware workflows
• Versioned eval output - Machine-readable eval results carry a schema version, so downstream automation stays stable across releases

• Python 3.11 or later - Lowered from 3.13, so the CLI installs in more development environments
• Default output filenames follow the chosen format - Breaking: Scan writes scan_<timestamp>.<format>, renamed from results_<timestamp>.json, and eval writes eval_<timestamp>.<format>. Update any automation that looks for the old results_*.json name

• More reliable adversarial evaluation - Built-in detector behavior tightened, with structured JSON judge output now required for key adversarial checks
• Smoother getting started - Clearer virtual environment setup and example-agent instructions, plus corrected command-reference links

• flintai scan - AI-powered static analysis of Python agent code, with findings mapped to the OWASP Top 10 for Agentic Applications and scored with CVSS severity
• flintai eval - Runtime behavioral evaluation of running agents using adversarial and functional tests
• Built-in evaluation content - Evaluations for OWASP LLM risks, plus the detector and model configuration needed to run them locally
• Framework detection - Auto-detect supported frameworks