About the company
Our client is a technology company.
The role
Raydar is recruiting for this role on behalf of our client. Design datasets, evaluation methods and measurement frameworks that shape how machine learning models are trained and assessed. Collaborate with external research partners, run fast experiments and convert findings into concrete specifications for large-scale training work.
What you'll do
- Design targeted data samples that reveal weaknesses in model behavior across a range of subject areas.
- Build and refine scoring rubrics and reward signals used in model training pipelines.
- Study how human annotators behave and run experiments to improve model capabilities.
- Develop quantitative methods for assessing dataset quality, diversity and downstream impact.
- Own the data pipelines that supply both naturally occurring and artificially generated training material.
- Work with research partners to turn their training goals into clear data and evaluation specifications.
Requirements
What we're looking for
- 1 to 4 years of experience in software or machine learning engineering.
- Strong Python skills and experience building dependable data pipelines, experimentation tooling or evaluation systems.
- Hands-on experience with language model evaluation, post-training, reward modeling or closely related ML infrastructure.
- Sound scientific judgment, including setting up well-controlled tests and drawing conclusions from imperfect results.
- Background in building or auditing training data and judging how well it supports model performance.
- Ability to learn unfamiliar domains quickly and reason across them.
Benefits
Compensation and benefits
- Base salary: USD 260,000 to 290,000 per year
- Equity
- Profit sharing
Location and work model
- San Francisco, CA, United States
- On-site
- Full-time