Production Reliability Engineer, AI Platform
3 hours ago
Winnipeg, Manitoba, Canada
Equinix
Full-time
Free with email or Google
Save this job and keep your search organized
Create a free account to save jobs, create alerts and return to this listing from your dashboard.
Free with email or Google
By continuing, you agree to our Terms & Privacy Policy.
Who are we?
Equinix is the world’s digital infrastructure company®, shortening the path to connectivity to enable the innovations that enrich our work, life and planet.
A place where tech thinkers and future builders turn bold ideas into breakthrough experiences, we welcome your unique perspective.Help us challenge assumptions, uncover bias, and remove barriers—because progress starts with fresh ideas. You’ll find belonging, purpose, and a team that welcomes you—because when you feel valued, you’re empowered to do your best work.
Job Summary Keeps Equinix's GenAI products and platforms stable and reliable, and makes them more resilient over time. Writes the code and automation that prevents failures, rather than only responding to them. Defines service-level objectives, builds monitoring and alerting, hardens services against dependency and capacity failures, and tests failure and recovery before they happen in production. Leads incident response when issues occur and turns each incident into a lasting fix. Works in a team with senior engineers and with product, platform and security partners to keep AI products, agents, connectors and self-hosted models robust, secure, observable and performant.
Knowledge and Application Applies solid knowledge of software engineering and site reliability engineering (SRE) practice: service-level objectives (SLOs), error budgets, monitoring, alerting, incident response, CI/CD and infrastructure as code. Understands how distributed systems fail and how to contain those failures (timeouts, retries, circuit breakers, graceful degradation). Understands how GenAI products behave in production, including model latency, rate limits, quotas, streaming and token cost. Follows established standards and suggests improvements.
Complexity and Problem Solving Works on problems of defined scope where analysis requires evaluation of identifiable factors. Diagnoses failures across application, model, data, pipeline and infrastructure layers, and fixes the underlying cause rather than the symptom. Applies existing runbooks and approaches, and improves them. Escalates unusual or high-impact issues. Work is reviewed at defined checkpoints.
Collaboration and Interaction Works with AI engineers, product managers, platform and security teams. Reviews designs and releases for reliability risk before they ship. Explains status, incidents and results clearly to partners who are not familiar with the subject. Asks for guidance early and shares knowledge with the team.
Responsibilities Reliability and resilience engineering (primary focus)
Write code and automation that makes GenAI products and platforms more resilient: retries, timeouts, circuit breakers, rate limiting, graceful degradation and fallback between models and providers
Run load tests, failure-injection tests and game days to find weak points before users do, and fix what they reveal
Harden services running on Kubernetes: health checks, autoscaling, disruption budgets, resource limits and multi-zone deployment
Plan capacity ahead of launches and growth, including model-provider quotas, GPU and inference capacity, and database headroom
Maintain backup, restore and disaster-recovery plans, and test recovery on a regular schedule
Automate repeated operational tasks, and reduce manual work each quarter
Monitoring and service levels
Define and track service-level indicators and objectives with product owners: availability, latency, error rate, token cost and quality scores
Instrument services with logs, metrics and traces (e.g., OpenTelemetry), from the user request through agent, tool and model calls
Build and tune dashboards and alerts for user-facing failures, and remove alerts that are noisy or not actionable
Release and deployment
Support release readiness for GenAI products and platforms: checks for monitoring, alerts, runbooks, capacity, security review and rollback plan
Deploy through CI/CD and GitOps pipelines, using containers and infrastructure as code, with canary rollouts and fast rollback
Incident response
Take part in the on-call rotation, respond to incidents, and write blameless post-incident reviews
Track post-incident actions to completion, so the same failure does not happen twice
Write and update runbooks, and turn recurring incidents into automation or fixes
GenAI and self-hosted models
Support the reliability of self-hosted open-weight and small language models: serving, GPU usage, scaling, rollout and rollback
Monitor model and provider latency, errors, rate limits and cost, and flag capacity and cost risks
Apply security, patching, secrets and data-boundary requirements to every deployment
Qualifications Required
1–4 years in site reliability, DevOps, platform o
A place where tech thinkers and future builders turn bold ideas into breakthrough experiences, we welcome your unique perspective.Help us challenge assumptions, uncover bias, and remove barriers—because progress starts with fresh ideas. You’ll find belonging, purpose, and a team that welcomes you—because when you feel valued, you’re empowered to do your best work.
Job Summary Keeps Equinix's GenAI products and platforms stable and reliable, and makes them more resilient over time. Writes the code and automation that prevents failures, rather than only responding to them. Defines service-level objectives, builds monitoring and alerting, hardens services against dependency and capacity failures, and tests failure and recovery before they happen in production. Leads incident response when issues occur and turns each incident into a lasting fix. Works in a team with senior engineers and with product, platform and security partners to keep AI products, agents, connectors and self-hosted models robust, secure, observable and performant.
Knowledge and Application Applies solid knowledge of software engineering and site reliability engineering (SRE) practice: service-level objectives (SLOs), error budgets, monitoring, alerting, incident response, CI/CD and infrastructure as code. Understands how distributed systems fail and how to contain those failures (timeouts, retries, circuit breakers, graceful degradation). Understands how GenAI products behave in production, including model latency, rate limits, quotas, streaming and token cost. Follows established standards and suggests improvements.
Complexity and Problem Solving Works on problems of defined scope where analysis requires evaluation of identifiable factors. Diagnoses failures across application, model, data, pipeline and infrastructure layers, and fixes the underlying cause rather than the symptom. Applies existing runbooks and approaches, and improves them. Escalates unusual or high-impact issues. Work is reviewed at defined checkpoints.
Collaboration and Interaction Works with AI engineers, product managers, platform and security teams. Reviews designs and releases for reliability risk before they ship. Explains status, incidents and results clearly to partners who are not familiar with the subject. Asks for guidance early and shares knowledge with the team.
Responsibilities Reliability and resilience engineering (primary focus)
Write code and automation that makes GenAI products and platforms more resilient: retries, timeouts, circuit breakers, rate limiting, graceful degradation and fallback between models and providers
Run load tests, failure-injection tests and game days to find weak points before users do, and fix what they reveal
Harden services running on Kubernetes: health checks, autoscaling, disruption budgets, resource limits and multi-zone deployment
Plan capacity ahead of launches and growth, including model-provider quotas, GPU and inference capacity, and database headroom
Maintain backup, restore and disaster-recovery plans, and test recovery on a regular schedule
Automate repeated operational tasks, and reduce manual work each quarter
Monitoring and service levels
Define and track service-level indicators and objectives with product owners: availability, latency, error rate, token cost and quality scores
Instrument services with logs, metrics and traces (e.g., OpenTelemetry), from the user request through agent, tool and model calls
Build and tune dashboards and alerts for user-facing failures, and remove alerts that are noisy or not actionable
Release and deployment
Support release readiness for GenAI products and platforms: checks for monitoring, alerts, runbooks, capacity, security review and rollback plan
Deploy through CI/CD and GitOps pipelines, using containers and infrastructure as code, with canary rollouts and fast rollback
Incident response
Take part in the on-call rotation, respond to incidents, and write blameless post-incident reviews
Track post-incident actions to completion, so the same failure does not happen twice
Write and update runbooks, and turn recurring incidents into automation or fixes
GenAI and self-hosted models
Support the reliability of self-hosted open-weight and small language models: serving, GPU usage, scaling, rollout and rollback
Monitor model and provider latency, errors, rate limits and cost, and flag capacity and cost risks
Apply security, patching, secrets and data-boundary requirements to every deployment
Qualifications Required
1–4 years in site reliability, DevOps, platform o