Documentation

AI deployment agent

The Gemini-powered deployment agent: how it works, configuration, limits and behaviour when Gemini is down.

Availability

AI Deploy is available when the Gemini API is configured and answering. /customer/ai shows its live status. While Gemini cannot answer, requests wait in the queue and running deployments are not affected.

How it works

Customers describe a deployment in plain language (/customer/ai) and authorize limits. A language model turns the request into a structured plan and diagnoses failures, using Google's Gemini API. Everything with consequences runs in the control plane's own code: catalog checks, limits, prices, wallet balance, node selection, capacity reservation, submission, verification, and stopping.

Component	Runs on	Responsible for
Control plane	The API (e.g. Render)	Auth, scheduling, reservations, billing, workflow state, audit, limit enforcement, and the calls to Gemini
Gemini API	Google	Answers planning and recovery questions with JSON
Compute nodes	Provider machines	Customer model deployments, through the existing worker

Every model call is queued first, so retries, expiry and cancellation do not depend on one HTTP call. The control plane answers each queued call right away with models/GEMINI_MODEL:generateContent; the API key travels in a header and never in a URL or log.

What the model decides, and what it does not

The model chooses among catalog models and recovery actions and writes explanations. It never decides authorization, GPU availability, prices, budget compliance, or whether a deployment works. It cannot call submission, rollback, recovery or health-check tools itself; it can only propose them, and the application validates every proposal.

Configure Gemini

Create an API key in Google AI Studio (https://aistudio.google.com/apikey).
Set it on the API as GEMINI_API_KEY (in the hosting platform's secret settings, never in the repository). Without it the agent is off and its endpoints answer ai_not_configured.
Optionally choose the model with GEMINI_MODEL (default gemini-2.5-flash). GEMINI_THINKING_BUDGET=0 (default) turns off thinking tokens, which would otherwise use up AI_MAX_TOKENS; unset it for models that cannot turn thinking off, such as Pro models.
Check GET /api/v1/ai/health: available with a key, not_configured without one, and degraded for a few minutes after Gemini fails to answer.

Setting	Default	Meaning
`GEMINI_TIMEOUT_SECONDS`	60	Per call; must not exceed AI_REQUEST_TIMEOUT.
`AI_MAX_ATTEMPTS`	2	Deliveries per model call. Timeouts, 429 and 5xx answers are retried on the next background pass (about every 15 s); other errors fail the run without a decision.
`AI_MAX_CONCURRENT_REQUESTS`	1	Gemini calls in flight at once, across all runs.

Customer requests go to Google

With Gemini, the customer's request text, the plan conversation and short tool results (catalog entries, deployment status and redacted log excerpts) are sent to Google for processing under the Gemini API terms. On Google's unpaid tier, Google may use that content to improve its products; use a paid (billing-enabled) project for production. Prompts sent to customers' own deployments never go to Gemini.

When Gemini is unavailable

The control plane, database, billing, and every running customer deployment keep working. Deployment state lives in the database, not in the model conversation.
A failed call (timeout, 429 or 5xx) is queued again and retried on the next background pass, up to AI_MAX_ATTEMPTS; the AI service shows as degraded meanwhile.
New AI requests are accepted and queued (up to AI_QUEUE_MAX_LENGTH). A call not answered within AI_QUEUE_TTL_SECONDS fails with “No AI decision was made”; nothing is deployed.
Deterministic steps continue: verification, duration and spend limits, stopping and cancellation all work without the model.

Autonomy limits

Customers set these per request; they are enforced in code, never by the prompt:

Maximum spend: the plan’s estimate must fit, the wallet must cover it, and the deployment is stopped when rated charges plus unrated READY time reach it.
Maximum duration: the deployment is stopped when it is reached.
Maximum CPU cores, memory, and GPU memory (0 = no GPU).
Maximum automatic recovery attempts (platform ceiling 3).
Allowed runtimes (Ollama, llama.cpp, vLLM) and model sources (Ollama registry, Hugging Face).
Agent steps and tool calls per run (platform settings).

Budget is checked, not held

The ledger has no holds, so “reserve budget” means: the wallet must cover the estimate when the deployment is created, node capacity is reserved atomically in the same transaction as the existing prepaid check, and spending is enforced by stopping the deployment. Another deployment could still use the same balance in the meantime.

Tests

cd server
pytest tests/test_ai_agent.py tests/test_ai_gemini.py     # against a fake Gemini API

The tests never call Google, never create paid deployments (no price is configured unless a test sets one), and never use GPUs. The fake Gemini checks every request the way the real API requires (key in the header, alternating turns, JSON responses).

Not yet verified

The Gemini integration has been tested against a fake that mirrors the documented generateContent API, not against Google itself; the first real call happens once GEMINI_API_KEY is set.
The quality of Gemini’s plans for this catalog has not been measured.
The end-to-end flow was tested with scripted model answers and a simulated worker tunnel, not with a real provider node running a model.