JOB-1544

A young, quick-moving applied research startup produces realistic training environments and datasets for advanced AI models. It needs a hands-on quality owner who decides what is good enough to ship, builds the automated systems that enforce that standard, and later assembles the team that works alongside them.

What you’ll own

  • Personally examine outputs from people-generated data and from environments, spending much of your time inside the data itself
  • Create automated evaluators, QA agents and scoring pipelines to detect reward hacking, faulty verifiers and mismatched data distributions ahead of delivery
  • Set measurable quality bars and pass criteria covering environments, tasks, verifiers, reward signals and datasets
  • Sign off on deliverables for engagements with leading AI labs, cloud providers and large enterprises
  • Recruit, onboard and manage quality reviewers and QA engineers while the group expands

Qualifications

  • One to five years as an LLM evaluation engineer, QA engineer or comparable role
  • Experience creating automated evaluators, QA tooling or data-quality systems for ML training data, agents or large pipelines
  • Technical understanding of what gives a dataset or environment real learning signal
  • Practice reviewing LLM or agent outputs and traces by hand, along with evals, RL environments or annotation pipelines
  • Ability to write the automation yourself: graders, judge models, QA agents and quality dashboards
  • Exacting attention to detail
  • Plus: past founder or zero-to-one experience

On-site in the San Francisco Bay Area. Visa transfers are considered.