ML Infrastructure Engineer (AI Agent Evaluation)

A startup that builds realistic environments and tasks for evaluating AI agents is hiring an infrastructure engineer. Its platform runs very large numbers of agents at the same time, each inside its own stateful environment, and this engineer will own how that work is dispatched and kept healthy.

Responsibilities

• Run the orchestration layer for a large fleet of simultaneous agent runs, including queue-based dispatch, concurrency limits and failure recovery
• Design environments and scenarios that feel real and test what matters rather than what is merely easy to measure
• Create new evaluations and extend support to new kinds of environments
• Build connections to third-party orchestration frameworks as the evaluation field changes quickly

Qualifications

• Background creating and operating LLM infrastructure and distributed systems
• Comfort with container orchestration such as Kubernetes and with message-driven designs using SQS or Kafka
• Demonstrated experience operating large-scale LLM workloads
• One to eight years in fast-moving engineering settings, either at startups or at scale-ups that grew meaningfully while you were there
• Stable tenure history

Hybrid in New York, NY or Seattle. Visa transfers considered.