Senior AI Agent Test Automation Engineer

7 days ago
Toronto ON, Toronto Census Division, ON; Ontario, Canada Nexxa Full-time
Own quality for Nexxa AI agents through Python automation and exploratory testing across industrial systems and operational scenarios. As a Senior or Staff SDET, you will build production-grade evaluation harnesses, simulators, regression suites, and CI/CD quality gates. You will test LLM-based agents across planning, tool selection, execution, recovery, and final outputs. The work combines API, contract, integration, end-to-end, performance, and reliability testing with manual trace review, red-teaming, and release verification.

Key Responsibilities

Build evaluation harnesses and regression suites for LLM-based agents Integrate automated evaluations into CI/CD quality gates Develop simulators, mocks, and sandbox environments for industrial tools Review traces, outputs, and agent behavior manually Turn failures and attacks into reproducible automated tests

Requirements:

Five-plus years in SDET, QA, or software engineering Strong Python programming and software engineering practices Manual and automated testing experience for distributed systems Experience with CI/CD, Docker, and Kubernetes Experience testing APIs, microservices, and LLM-based products Apply your automation, evaluation, and exploratory testing expertise to Nexxa’s high-stakes industrial AI systems.

#J-18808-Ljbffr

  • Senior AI Agent Test Automation Engineer

    6 days ago

    Toronto, Ontario, Canada Nexxa Full-time

    Own quality for Nexxa AI agents through Python automation and exploratory testing across industrial systems and operational scenarios.As a Senior or Staff SDET, you will build production-grade evaluation harnesses, simulators, regression suites, and CI/CD quality gates. You will test LLM-based agents across planning, tool selection, execution, recovery, and...

  • AI-Driven Senior Test Automation Engineering

    2 days ago

    toronto, on, Canada PureFacts Financial Solutions Full-time

    Excel as a Senior Test Automation Engineer at PureFacts, specializing in AI-driven testing methodologies in Toronto. Transform software quality and efficiency with AI technologies.In this role, you will join the Quality Engineering team to innovate and optimize testing strategies. Your extensive experience in test automation, AI tools, and database...

  • Senior Test Automation Engineer

    1 day ago

    Toronto, ON, Canada PureFacts Financial Solutions Full-time €100,000 - €120,000 Contract

    Senior Test Automation Engineer PureFacts Financial Solutions Toronto, Ontario, Canada About this position About PureFacts Financial Solutions PureFacts is the leader in the Revenue Performance Management category for wealth and asset management firms. The PureRevenue™ Platform helps organizations maximize revenue potential by connecting pricing,...

  • Senior Automation Testing Engineer

    3 days ago

    Mississauga, ON, Peel Regional District, ON; Ontario, Canada Novozen Healthcare LLC Full-time

    Job Title: Senior Automation Engineer with DeltaV This position is open exclusively to people who are eligible to work in USA Location : United States Job Summary We are seeking a Senior Automation Engineer with DeltaV expertise to support commissioning and qualification (C&Q) activities in a biopharma manufacturing environment. This role requires...

  • Senior Agentic AI Engineer

    5 days ago

    Toronto, Ontario, Canada enableIT Full-time

    Senior Agentic AI Engineer Location: Toronto, ON Work Arrangement: Hybrid – 50% Remote Employment Type: Long term contract Interview: 3 rounds Job Description: We are seeking a Senior Agentic AI Engineer to design, develop, and deploy production-grade AI applications using Large Language Models (LLMs), Agentic AI frameworks, Retrieval-Augmented...

  • Senior Test Automation Engineer – AI

    4 days ago

    Toronto, ON, Canada Source Code Full-time $96,000 - $117,000 Contract

    Source Code is seeking a Sr. Automation Developer to work onsite at 5700 Yonge Street in Toronto. The role focuses on identifying automation opportunities, designing and executing QA tests with automation scripts, and developing test automation frameworks.Ideal candidates have 7+ years in automation testing, experience with LoadRunner and Selenium, and a...

  • Senior Qa Automation Engineer – Testing

    1 day ago

    toronto, ontario, Canada Rubicon Path Full-time

    A leading tech firm in Toronto, Canada is seeking a Senior Automation Developer to enhance software automation processes. Key responsibilities include designing QA tests, executing test automation scripts, and collaborating with development teams. Ideal candidates will have over 10 years of experience in testing and be proficient with tools like Katalon and...

  • AI Solutions Engineer, Agentic Automation

    3 days ago

    Toronto, ON, Toronto Census Division, ON; Ontario, Canada Urban Ridge Supplies Full-time

    About Northleaf Capital Partners Northleaf is a global private markets investment firm focused on mid-market companies and assets. With US$31 billion in capital commitments raised to date, Northleaf has an established, long-term track record as an investor in private equity, private credit and infrastructure globally. Northleaf's 300+ person team, located...

  • Senior AI Test Automation Developer

    2 days ago

    toronto, on, Canada Vena Full-time

    Build reliable AI product testing as a Senior Software Developer in Test with Vena, working remotely from Canada on automated quality solutions. This full-time senior role supports Vena’s agentic AI solutions within Product Development. You will develop, execute, and maintain automated tests while defining approaches for AI-powered products. The work...

  • Senior AI Test Automation Developer

    2 days ago

    toronto, on, Canada Vena Solutions Inc. Full-time

    Build reliable AI product testing as a Senior Software Developer in Test with Vena, working remotely from Canada on automated quality solutions. This full-time senior role supports Vena’s agentic AI solutions within Product Development. You will develop, execute, and maintain automated tests while defining approaches for AI-powered products. The work...

Similar jobs in Toronto ON

Top cities