Quickstart: Optimize a hosted agent (preview)

In this quickstart, you deploy the optimization sample agent, run the agent optimizer to improve its instructions, and deploy the winning candidate.

Prerequisites

Before you begin, you need:

• An Azure subscription--Create one for free.

• azd CLI (Azure Developer CLI).

• Azure CLI for authentication.

• The microsoft.foundry extension for azd (0.1.40-preview or later of the azure.ai.agents dependency):

  azd ext install microsoft.foundry
  

  If already installed, upgrade:

  azd ext upgrade microsoft.foundry
  

• Your Azure subscription must be on the allow list for the agent optimizer. Contact your Microsoft representative to request access.

Step 1: Create the project

Initialize a new project from the optimization sample template:

mkdir my-agent && cd my-agent
azd ai agent init -m https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/bring-your-own/responses/optimization-customer-support/agent.manifest.yaml .

The interactive flow prompts for your Azure subscription, region, and model deployment settings. It generates agent.yaml, .agent_configs/baseline/, the evaluation dataset, and infrastructure files.

Step 2: Provision and deploy

Authenticate and provision the Azure resources:

az login
azd auth login
azd provision

Provisioning takes approximately two minutes and creates a Foundry account, project, Azure Container Registry, and model deployments.

Deploy the agent:

azd deploy

Test the deployment:

azd ai agent invoke "What is 2+2?"

Step 3: Generate evaluation suite and optimize

Generate an evaluation dataset and evaluators for your agent:

azd ai agent eval generate

This step creates eval.yaml, a test dataset, and scoring evaluators based on your agent's instructions. The optimizer uses these files to measure improvement.

Run the optimizer:

azd ai agent optimize --max-candidates 2

The CLI prompts you to select an optimization model. To skip the prompt, pass it directly:

azd ai agent optimize --max-candidates 2 --optimize-model gpt-5

The CLI detects your agent from agent.yaml and uses the generated eval.yaml automatically. With two candidates, optimization typically completes in about 8 minutes. Real-time progress is shown:

Optimizing agent "customer-support-py"...
  Config: eval.yaml
  Baseline saved to .agent_configs/baseline/metadata.yaml
  Job ID: opt_162bd0f09....
  Status: pending
  Portal: <OPTIMIZATION-JOB-URL>

Use the portal URL to monitor your job in the Foundry portal.

The eval model scores each response (any chat-completion model works). The optimization model (--optimize-model) generates improved candidates and must be from the supported list (gpt-5 family or DeepSeek). You can also set optimization_model under options: in eval.yaml to avoid passing the flag each time.

Step 4: Deploy the winner

The ★ in the output indicates the best candidate. Apply the optimized config locally, then deploy:

azd ai agent optimize apply --candidate <candidate-id>
azd deploy

The apply command downloads the optimized configuration into .agent_configs/<candidate_id>/ and updates your agent.yaml to use the new instructions. The deploy command pushes the optimized agent live using code deploy.

Invoke your agent to verify the improvement:

azd ai agent invoke "What is your return policy?"

You can also run evaluation to confirm the score improvement:

azd ai agent eval run

Clean up resources

When you finish experimenting, delete the provisioned resources:

azd down --force --purge

Troubleshooting

What you learned

In this quickstart, you:

• Deployed the optimization sample agent by using the customer-support template.
• Ran the agent optimizer to automatically improve agent instructions.
• Deployed the winning candidate and verified the improvement.

Next steps

• Agent optimizer overview
• Create a custom evaluation dataset
• Optimize agent instructions, skills, tools, and models
• Make your agent optimizer-ready
• Run agent evaluations with the azd CLI