Software Development Engineer in Test

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Development Engineer in Test - Agentic Automation based in Canada.

This is an opportunity for an experienced SDET to help shape how AI agents are tested, evaluated, and trusted in production. You will join a platform engineering team working on a large-scale, cloud-based cybersecurity environment, building the automation that supports reliability, performance, and quality. Alongside traditional functional and integration testing, you will contribute to agentic test generation, AI-powered workflows, and eval-driven development. Your work will help ensure that AI systems behave accurately, safely, and consistently before reaching production. You will collaborate with engineers across the development lifecycle in a remote-first environment that values autonomy, experimentation, and continuous improvement.

Accountabilities

• Design, develop, and maintain automated test suites and frameworks for distributed, cloud-based systems.
• Maintain and extend agentic test generation and analysis capabilities within core testing frameworks.
• Build AI-powered workflows and agent skills that reduce manual testing effort, including test data setup, failure triage, and release validation.
• Maintain evaluation suites, golden datasets, and scoring pipelines to assess AI agents across correctness, tool usage, execution trajectories, faithfulness, safety, latency, and cost.
• Implement CI evaluation gates to measure changes to prompts, models, tools, and orchestration before release, incorporating production failures into future evaluations.
• Identify and mitigate unreliable AI outputs, including hallucinated assertions, flaky or redundant tests, inaccurate failure analysis, and evaluation bias.
• Design and execute performance, load, and stress tests to identify bottlenecks and validate the scalability of distributed services.
• Test integrations across platform systems, services, data, applications, and customer-facing features.
• Review product, system, and agent designs to assess testability, evaluation coverage, and opportunities for improvement.
• Share technical knowledge and contribute to team-wide quality and evaluation standards.

Requirements

• Degree in Computer Science or equivalent practical experience, with a solid understanding of object-oriented programming, data structures, algorithms, and distributed systems.
• At least 5 years of experience in software quality, including 3+ years of hands-on SDET or test automation development.
• Strong programming skills in at least one of Java, Node.js, or Python; Python is preferred for AI evaluation and agent-related work.
• Demonstrated experience designing, developing, and maintaining test frameworks integrated into CI/CD pipelines such as Jenkins or GitLab CI/CD.
• Experience testing web services and APIs, including REST, gRPC, and JSON.
• Hands-on performance and load testing experience with tools such as Locust or k6, including workload design and interpretation of latency percentiles, throughput, and error rates.
• A skeptical, verification-first approach to AI-generated output, with the ability to distinguish when deterministic assertions are appropriate and when probabilistic evaluation is needed.
• Experience applying AI technologies to improve decision-making, streamline workflows, increase efficiency, and deliver business outcomes.
• Strong analytical, problem-solving, and collaboration skills, with the ability to communicate technical findings and contribute to shared quality standards.

Nice to have

• End-to-end eval-driven development experience, including release gating, trajectory and tool-use evaluations, and calibrating LLM-as-judge systems against human review.
• Experience with prompt engineering and designing agent context or tools.
• Experience integrating performance tests into CI/CD pipelines and using observability tools such as Grafana, Prometheus, or CloudWatch.
• Background in cybersecurity, threat intelligence, malware analysis, or sandboxing and detonation environments.
• Familiarity with technologies such as MongoDB, MinIO, RabbitMQ, Kafka, AWS services (EC2, S3, DynamoDB), Docker, Nomad, Terraform, Unix/Linux, Git, Jenkins, GitLab CI/CD, Jira, and Bitbucket.
• Experience with LLM APIs, agent frameworks, and LLM evaluation tooling.

Benefits

• Annual base salary ranging from CA$120,000 to CA$170,000, plus variable or incentive compensation, equity, and benefits.
• Market-leading compensation and equity award opportunities.
• Comprehensive physical and mental wellness programs.
• Competitive vacation allowance an