JOB-1720
A research-driven startup building a memory and personalization layer for AI agents is hiring an engineer to take full ownership of how its system is measured. The product is an intricate system of cooperating agents with partly opaque behavior, and the central puzzle is deciding what improvement even means for representations of people and entities that evolve over time and have no ground truth.
Focus areas
- Design the evaluations: define scores, work out what improvement actually looks like for an evolving representation, and revisit that definition as the product evolves
- Construct the data pipelines and test harnesses behind them, covering data intake, labeling, versioning, reruns and automated judges, so new questions get answers within days
- Run the evaluations, read traces and results, and identify what is really broken rather than merely easy to measure, then propose fixes
- Create large-scale agent simulations, because the pace of iteration depends on the number of entities that can be modeled simultaneously
- Own the loop from question to pipeline to rerun to writeup, with no handoffs
Credentials
- Three or more years designing and building LLM evaluations, including complex evals, simulators or environments for RL post-training
- Evidence of strong research: first-author or co-author work at a top ML venue, or open-source work of comparable quality
- Production-quality Python, with tests and sensible release habits
- Familiarity with PyTorch, Hugging Face and experiment tracking; data pipeline tools such as Kafka or SQL are a plus
- Experience with synthetic users, memory or personalization for LLMs is a bonus
On-site in New York, NY. No visa sponsorship; TN visas can be accommodated, H-1B cannot.
