JOB-1529
A fast-growing Series B company that turns web pages into clean data for AI systems wants an engineer to build the evaluation layer that shows whether its output is actually correct. Judging quality across enormous numbers of sites, layouts and odd edge cases is a hard measurement problem, and the person in this seat will define how it gets solved.
What you’ll own
- Define the metrics that capture what a good result looks like across a huge variety of websites and formats
- Build the test harnesses and scoring flows that measure quality with rigor and at volume
- Produce and maintain data collections that make those measurements trustworthy
- Close the loop between measured output quality and decisions about models and product
- Present results in a form the entire team can act upon
About you
- Four or more years in ML, research engineering or data-heavy backend work, with real evaluation experience
- Has built evals for judging LLM-based extraction, such as checking whether the right fields were pulled from messy pages
- Designed the metrics and the evaluation system yourself rather than running someone else’s benchmark
- Built ground-truth datasets with gold labels or reference extractions
- Statistical rigor: significance, estimator bias and variance, sampling error, and agreement between automated judges and humans
- Strong Python, and comfort with problems where you must invent the yardstick
- A bias for shipping, measuring and iterating
- Eval experience limited to agents, recommendation ranking, search, safety work or public model leaderboards is not what we need
Hybrid in the San Francisco Bay Area, with three or more days in the office. Visa transfers and certain work authorizations supported; new sponsorship is not available.
