Machine Learning Evaluation Specialist
Save this job and keep your search organized
Create a free account to save jobs, create alerts and return to this listing from your dashboard.
By continuing, you agree to our Terms & Privacy Policy.
About The Role
What if your years of hard-earned research expertise could directly shape the future of AI? We're looking for domain experts with deep machine learning knowledge to design evaluation challenges that push state-of-the-art AI systems to their limits — the kind of problems only a true specialist could craft. Your work won't sit in a drawer. It directly influences how the next generation of AI models are measured, trained, and improved.
• Organization: Alignerr
• Type: Hourly Contract
•
Location:
Remote
• Commitment: 10–40 hours/week
What You'll Do
• Design complex, original machine learning problems rooted in your area of domain expertise
• Create evaluation tasks that demand advanced knowledge well beyond standard ML pipelines
• Draw from your own research experience to craft challenges that genuinely test highly capable AI models
• Write clear problem statements, define evaluation criteria, and establish gold-standard solutions
• Assess AI-generated solutions for correctness, creativity, and methodological rigor
• Document problem difficulty, required domain knowledge, and expected failure modes
• Collaborate asynchronously with a global team of researchers and engineers Who You Are
• Graduate-level expertise (MS or PhD preferred) in a scientific or technical discipline that intersects with machine learning
• Strong working knowledge of ML methods — model selection, feature engineering, evaluation metrics, and pipeline design
• Deep familiarity with active, open research problems in your field
• A sharp eye for where general ML knowledge breaks down and specialized domain insight becomes essential
• Experience publishing or conducting original research is highly valued
• Excellent written communication — you can articulate complex, nuanced problems with precision and clarity
• Self-motivated and energized by intellectually demanding, independent work Example Domains We welcome experts from a wide range of fields, including but not limited to:
• Computational biology, genomics, or bioinformatics
• Climate science and environmental modeling
• Medical imaging and healthcare ML
• Materials science and computational chemistry
• Astrophysics and signal processing
• Natural language processing for low-resource or specialized corpora
• Robotics, control theory, or reinforcement learning in complex environments
• Financial modeling and quantitative analysis If your domain sits at the frontier of ML research, we want to hear from you. Why Join Us
• Work at the cutting edge — your challenges help define the boundaries of what AI can and cannot do
• Make a real impact — your expertise directly shapes AI safety and evaluation research
• Full autonomy — work on your own schedule, from anywhere in the world
• Flexible commitment — scale hours up or down based on your availability
• Ongoing opportunity — strong contributors are considered for contract extensions and deeper research involvement
• Build
your profile
— establish yourself as a contributor to frontier AI development alongside top research labs
-
Machine Learning Evaluation Specialist
5 hours ago
Toronto, Ontario, Canada Alignerr Corp. Full-timeAbout The RoleWhat if your years of hard-earned research expertise could directly shape the future of AI? We're looking for domain experts with deep machine learning knowledge to design evaluation challenges that push state-of-the-art AI systems to their limits — the kind of problems only a true specialist could craft. About The RoleWhat if your years of...
-
Machine Learning Evaluation Specialist
1 week ago
Toronto, ON, Toronto Census Division, ON; Ontario, Canada Alignerr Corp. Full-timeAbout The RoleWhat if your years of hard-earned research expertise could directly shape the future of AI? We're looking for domain experts with deep machine learning knowledge to design evaluation challenges that push state-of-the-art AI systems to their limits — the kind of problems only a true specialist could craft. About The Role What if your...
-
Borealis AI Machine Learning Specialist
5 hours ago
Toronto, Ontario, Canada Borealis AI Full-timeBecome a Machine Learning Specialist at Borealis AI and shape the future of AI through innovative research. Channel your skills in Python and deep learning to develop groundbreaking models. As a valuable team member at Borealis AI, you will engage with extensive datasets to explore and accelerate machine learning research. Candidates holding a PhD or those...
-
Pinterest Senior Machine Learning Specialist
5 hours ago
Toronto, Ontario, Canada Pinterest Full-timeJoin Pinterest as a Senior Machine Learning Specialist and enhance user experiences with your expertise in deep learning. Help us innovate through the power of AI. We are seeking a Senior Machine Learning Engineer with over four years of industry experience at Pinterest. This role entails the development of machine learning models that drive improvements...
-
Machine Learning Engineer
12 hours ago
Toronto, Ontario, Canada Alignerr Remote Full-time US$50 - US$70 PermanentMachine Learning Engineer (AI Data Trainer) About The Role What if your deep technical expertise could directly shape how the next generation of AI models reason, plan, and solve real-world problems? We're looking for Machine Learning Engineers to help train and improve large language models by demonstrating how expert-level thinking actually works —...
-
AI Evaluation Scientist
22 hours ago
Toronto, Ontario, Canada Vector Institute Full-time $125,800 - $157,300 ContractAI Evaluation Scientist, Learning and Cognition Position Summary We are seeking an AI Evaluation Scientist, Learning and Cognition to design and implement evaluation methods for AI learning agents to be used in the post-secondary education sector. This Scientist role focuses on establishing criteria for high-quality learning agents and how they should be...
-
Senior Software Developer
17 hours ago
Toronto, Ontario, Canada Vector Institute Full-time $125,800 - $157,300 ContractSenior Software Developer, Machine Learning POSITION SUMMARY As a Senior Software Developer, Machine Learning, your primary focus will be on packaging machine learning research into reusable software artifacts. You will collaborate with cross-functional teams, including researchers, software developers, and product managers, to design, develop, and deploy...
-
Senior Machine Learning Engineer
2 days ago
Toronto, Ontario, Canada TD Full-time $115,600 - $163,200 ContractWork Location: Toronto, Ontario, Canada Hours: 37.5 Line of Business: Analytics, Insights, & Artificial Intelligence Pay Details: $115,600 - $163,200 CAD This role is temporarily eligible for a pay premium above the posted salary range that is reassessed annually. You are encouraged to have an open dialogue with your recruiter who can provide more specific...
-
Software Engineer – Machine Learning
12 hours ago
Toronto, Ontario, Canada Alignerr Remote Full-time US$90 - US$120 PermanentAbout The Role What if your machine learning expertise could directly influence how the most advanced AI systems in the world think, reason, and write code? We're looking for experienced ML engineers in Toronto to evaluate AI-generated machine learning solutions — catching errors, improving quality, and helping frontier AI models get genuinely better at...
-
Applied Machine Learning Scientist II
2 days ago
Toronto, Ontario, Canada TD Full-time $125,000 - $154,000 ContractWork Location: Toronto, Ontario, Canada Hours: 37.5 Line of Business: Analytics, Insights, & Artificial Intelligence Pay Details: $125,000 - $154,000 CAD The pay details posted reflect a temporary market premium specific to this role that is reassessed annually. TD is committed to providing fair and equitable compensation opportunities to all colleagues....
