Manager, Generative AI Applied ML

Scale's Generative AI Data Engine powers the most advanced LLMs and generative models in the world through RLHF/RLAIF, data generation, model evaluation, safety, and alignment.

As the Manager of the Generative AI Applied ML team, you will lead a talented team of research engineers and ML engineers focused on delivering scalable, production-ready solutions to support Scale's GenAI Data Engine, such as rater-assistant models, LLM as a judge, critique modeling, fraud and cheating detection, etc. This role is critical for designing and executing a roadmap that accelerates our frontier model building customers' Generative AI initiatives forward. We are looking for someone with a strong background and hands-on experience in fine-tuning and evaluating LLMs, prioritizing practical, production-oriented problem-solving over academic research. This position requires a deep commitment to building robust, efficient systems that meet the demands of large-scale production, supporting millions of tasks monthly across a hybrid human-machine system, with the aim to scale into billions monthly.

This is a unique opportunity to drive impactful, production-focused work on the frontier of AI, collaborating with industry-leading professionals to shape and deliver reliable, high-performance solutions for real-world applications.

• Manage a team of highly effective research engineers and ML engineers. Provide guidance, mentorship, and technical leadership to a team of researchers and engineers working on Generative AI projects.
• Develop and evaluate methods for integrating machine learning into human-in-the-loop labeling systems to ensure high-quality and throughput labels for our customers.
• Implement and improve on state-of-the-art research developed internally and from the community and put them into production to solve problems for our customers and taskers.
• Work with product and research teams to identify opportunities for improvement in our current product line and for enabling upcoming product lines.
• Work with massive datasets to develop both generic models as well as fine-tune models for specific products.
• Build a scalable autorating platform that will improve quality and efficiency of the generative AI data engine.
• Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results.
• Must be able to commute to the San Francisco Office 2+ times weekly.

• 7+ years of full time work experience in deep learning, deep reinforcement learning, or natural language processing in a production environment, especially post-training experience with LLMs.
• Experience managing a 5-20 people team or leading a technical workstream.
• Strong programming skills in Python, experience in PyTorch or Tensorflow
• Experience with MLOps and the automation of model training & evaluation
• Experience working with cloud technology stack (eg. AWS or GCP) and developing machine learning models in a cloud environment
• Deep appreciation for building high-quality, robust, reusable productionized ML systems.

• Publication experience in the field or related topics.
• Experience with using LLMs as a judge for AutoRating systems.
• Graduate degree in Computer Science, Machine Learning or Artificial Intelligence specialization