About the Role
Join an AI/ML team building tools to make production AI agents more reliable. You will own the intelligence that identifies agent failures, assesses confidence, and checks whether proposed fixes work.
What You'll Do
Detect subtle failures, including omissions, compliant but incorrect responses, and patterns that emerge across large volumes of traces.
Turn user behavior signals, such as rephrasing, abandonment, and retries, into evidence of agent failures.
Build evaluation methods to measure detection precision when labeled ground truth is limited or unavailable.
Make patch generation trustworthy by reproducing failures, verifying proposed fixes, and avoiding low-confidence changes.
Improve the cost efficiency of detection, choosing appropriate models and approaches for each event.
Review production customer traces regularly to identify problems and opportunities for improvement.
What We're Looking For
Demonstrated AI/ML engineering experience; candidates with 0 to 10 years of experience are welcome, including new graduates and people without a traditional degree.
Hands-on experience using LLMs in production, including evaluations, LLM-as-judge, embeddings, and deciding when a smaller model or no model is appropriate.
An individual-contributor engineering background and a track record of shipping products that people use.
Strong research judgment and the ability to distinguish meaningful improvements from noise, especially when ground truth is unavailable.
Experience running agents in production and learning from their failures is a plus.
Compensation & Benefits
Base salary: $120,000 to $200,000 annually. Visa sponsorship is not available.
Location
On-site in San Francisco, United States.