Senior AI Observability Engineer

16 hours ago
Toronto, ON, Canada Aimlroles Full-time €80,000 - €175,000/year
Build AI observability systems at BMO in a hybrid role, using Python, telemetry, and evaluation methods for production models and agents. As a Senior AI Engineer, you will design systems that trace LLM calls, monitor token usage, detect drift, and evaluate autonomous workflows. The work includes machine learning, deep learning, statistical analysis, and specialized technical support. You will help govern agent behavior and diagnose non-routine production problems.

Key Responsibilities
- Design and maintain AI model observability systems
- Trace agent trajectories, prompts, tools, and outputs
- Build automated evaluations for relevance and hallucinations
- Monitor drift, latency, errors, and data quality
- Define guardrails and self-remediation boundaries

Requirements:
- Bachelor's degree in Computer Science or Engineering
- Seven to ten years of Site Reliability Engineering experience
- Experience in Artificial Intelligence Observability Engineering
- Python, SQL, LogQL, or PromQL proficiency
- Experience with Prometheus, Grafana, or Datadog Apply AI engineering, observability, and evaluation expertise to support reliable agent workflows at BMO.

  • Senior AI Observability Engineer

    16 hours ago

    Toronto, ON, Canada Aimlroles Full-time

    Build AI observability systems at BMO in a hybrid role, using Python, telemetry, and evaluation methods for production models and agents. As a Senior AI Engineer, you will design systems that trace LLM calls, monitor token usage, detect drift, and evaluate autonomous workflows. The work includes machine learning, deep learning, statistical analysis, and...

  • Senior AI Engineer

    2 days ago

    Toronto, ON, Canada BMO Financial Group Full-time

    Groupe BMO recrute un spécialiste en apprentissage automatique et IA pour concevoir et optimiser des systèmes ML dans un cadre hybride à Toronto, Canada. Vous travaillerez avec les domaines de données et d’analytique pour livrer des solutions reproductibles et décisionnelles. Le rôle exige des compétences en ML/Deep Learning, tests, et collaboration...

  • Senior AI Engineer for Observability

    16 hours ago

    Toronto, ON, Canada Aimlroles Full-time €80,000 - €175,000 Contract

    Application Deadline: 11/29/2026 Address: 33 Dundas Street West Job Family Group: Data Analytics & Reporting Hybrid role Designs, builds, and maintains systems that track, evaluate, and debug non-deterministic AI models and agent workflows. Trace LLM calls, monitor token usage, detect drift, set up automated evaluations, and govern autonomous agent...

  • Senior AI Engineer for Observability

    1 day ago

    toronto, on, Canada BMO Full-time

    Address: 33 Dundas Street West Job Family Group: Data Analytics & Reporting Hybrid role Designs, builds, and maintains systems that track, evaluate, and debug non-deterministic AI models and agent workflows. Trace LLM calls, monitor token usage, detect drift, set up automated evaluations, and govern autonomous agent behaviors in production. Designs and...

  • Senior ai engineer for observability

    2 days ago

    Toronto, ON, Canada BMO Financial Group Full-time

    Date limite pour présenter sa candidature :11/29/2026Adresse :33 Dundas Street WestGroupe de famille d’emploi :Analyses des données et communication de l'informationHybrid roleEffectuer des recherches sur des systèmes d’intelligence artificielle évolutifs capables d’apprendre et de faire des prédictions quant aux exigences opérationnelles, et...

  • Senior Observability Engineer

    16 hours ago

    Toronto, ON, Canada CI Financial Corp. Full-time $85,000 - $125,000 Contract

    At CI, we see a great place to work as one that is a safe place for everyone to have a voice, where people are empowered to take ownership over meaningful work, where there is an opportunity to grow through stretching themselves, where they can work on innovative products and projects, and where employees are supported and engaged in doing so.We are seeking...

  • Senior Observability Engineer

    2 days ago

    Toronto, ON, Canada CI Financial Full-time

    At CI, we see a great place to work as one that is a safe place for everyone to have a voice, where people are empowered to take ownership over meaningful work, where there is an opportunity to grow through stretching themselves, where they can work on innovative products and projects, and where employees are supported and engaged in doing so.We are seeking...

  • Senior Support Engineer in Observability

    4 days ago

    Toronto, ON, Canada Tata Consultancy Services Full-time

    Tata Consultancy Services seeks a Senior Support Engineer specializing in observability. Drive system resilience using Terraform and advanced monitoring tools in your role.As a Senior Support Engineer, you'll be pivotal in implementing robust observability solutions. Your hands-on experience will help drive system improvements, leveraging tools like...

  • Senior Support Engineer in Observability

    4 days ago

    Toronto, ON, Toronto Census Division, ON; Ontario, Canada Tata Consultancy Services Full-time

    Tata Consultancy Services seeks a Senior Support Engineer specializing in observability. Drive system resilience using Terraform and advanced monitoring tools in your role. As a Senior Support Engineer, you'll be pivotal in implementing robust observability solutions. Your hands-on experience will help drive system improvements, leveraging tools like...

  • Senior Software Engineer, Observability

    3 days ago

    Toronto, ON, Canada Okta Full-time

    Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate...

Similar jobs in Toronto

Top cities