Principal AI Cloud Engineer – Full Stack

Application Deadline:

Address:

Job Family Group:

We are back in office 2-4 days/week This role is not remote/virtual.

The Team

We accelerate BMO's AI journey by building enterprise-grade, cloud-native AI solutions. Our team combines engineering excellence with cutting-edge AI to deliver scalable, secure, and responsible solutions that power business innovation across the bank. We enable and accelerate our partners on their AI journeys across the enterprise, helping teams across BMO unlock value at scale. We support one another in times of need and take pride in our work. We are engineers, AI practitioners, platform builders, thought leaders, multipliers, and coders. Above all, we are a global team of diverse individuals who enjoy working together to create smart, secure, and scalable solutions that make an impact across the enterprise. Our ambition is bold: deploy our capital and resources to their highest and most profitable use through a digital-first operating model, powered by data and AI-driven decisions.

The Impact

As a Principal Cloud AI Engineer- Full Stack, you are a hands-on technical developer who designs, builds, and scales cloud-native AI solutions and products. You help set engineering standards, establish patterns, mentor senior engineers, and partner with multiple teams to deliver resilient, governed, and cost-efficient AI at enterprise scale.

You'll help shape and evolve our AI cloud strategy from model serving and LLMOps to security, observability, and compliance so teams across the bank can innovate safely and rapidly.

You will advance BMO's Digital First strategy by:

• Defining reference and production-grade solutions for AI/GenAI on cloud (Azure preferred; multi-cloud aware).

• Building reusable, secure, and observable components (APIs, SDKs, microservices, pipelines).

• Operationalizing LLMs and RAG with strong controls and Responsible AI guardrails.

• Driving platform roadmaps that enable faster delivery, lower risk, and measurable business outcomes.

What's In It for You

• Influence the technical direction of enterprise AI and the platform primitives others build on.

• Ship high-impact systems used across many business lines and products.

• Work across the full stack: cloud infra, data/feature pipelines, model serving, LLMOps, and DevSecOps.

• Partner with a leadership team invested in your growth and thought leadership.

Responsibilities

Product Builder

• Build and operate AI/ML cloud-native systems: frontend, backend, integration to other systems, feature stores, training/serving infra, vector databases, model registries, CI/CD, canary/blue-green, and GitOps for AI.

• Technical cloud-native implementation of ML/LLM observability (latency, cost, drift, hallucination/guardrails, quality & safety metrics), logging/tracing (OpenTelemetry), and SLOs/SLIs for production AI systems.

• Design and implement robust CI/CD pipelines for AI/ML workloads using GitHub Actions and Azure DevOps, including automated testing, model validation, security scanning, model versioning, and blue/green or canary deployments to ensure safe, repeatable, and auditable releases.

• Drive FinOps for AI/GPU workloads (rightsizing, autoscaling, spot, caching, inference optimization).

Strategy

• Help evolve the cloud AI reference design (networking, security, data, serving, observability) for ML/GenAI workloads (batch, streaming, online) with HA/DR, multi-region patterns, and cost efficiency.

• Work on standards and best practices for containerization, microservices, serverless, event-driven design, and API management for AI systems.

GenAI & LLMOps

• Architect RAG systems (chunking, embeddings, vector stores, grounding, evaluation) and guardrail frameworks (prompt/content safety, PII redaction, jailbreak & injection defenses).

• Lead model serving (LLMs and traditional ML) using performant runtimes (e.g., TensorRT-LLM, vLLM, Triton/KServe) and caching strategies; optimize token usage, throughput, and cost.

• Guide fine-tuning/PEFT/LoRA strategies, evaluation frameworks (offline/online A/B), and safety/quality scorecards; standardize prompt libraries and prompt engineering patterns.

Security, Risk & Governance

• Implement defense-in-depth: IAM least privilege, private networking, KMS/Key Vault, secrets mgmt, image signing/SBOM, policy-as-code (OPA/Azure Policy), and data sovereignty controls.

• Embed Responsible AI: model documentation, lineage, explainability, fairness testing, and human-in-the-loop patterns; align to model risk management and audit needs.

• Ensure regulatory and privacy compliance (e.g., PII handling, encryption in transit/at rest, approved data sources, retention & residency).

Delivery & Operations

• Lead complex discovery and solution design with stakeholders; build strong business cases (value, feasibility, ROI).

• Oversee production readiness and operate platforms with SRE principles (SLOs, error budgets, incident response, chaos testing, playbooks).

• Mentor engineers; multiply team impact via reusable components, templates, and inner-source.

Qualifications

Must Have

• Bachelor's, Master's, or PhD in Computer Science, Engineering, Mathematics, or related field (or equivalent experience).

• 7+ years building large-scale distributed cloud systems; 5+ years hands-on with cloud (Azure preferred; AWS/GCP nice to have).

• Proven experience designing and operating production ML/GenAI systems (training, serving, monitoring) and shipping AI features at scale on cloud.

• Strong software engineering in Python (and one of Go/Java/TypeScript); deep expertise with APIs, async patterns, and performance optimization.

• Hands-on with MLOps/LLMOps: MLflow, KServe/Triton, Feast/feature stores, vector DBs (e.g., FAISS, Milvus, Pinecone, pgvector, Cosmos DB with vectors), orchestration (Airflow/Prefect), and CI/CD for ML (GitHub Actions/Azure DevOps).

• Cloud-native stack: Kubernetes (AKS/EKS), containers, service mesh/ingress, serverless (Azure Functions/Lambda), IaC (Terraform/Bicep), secrets & key management, VNet/Private Link/peering.

• GenAI production experience: RAG, evaluation, prompt engineering, fine-tuning/PEFT/LoRA, and integration with providers (e.g., Azure OpenAI/OpenAI, Anthropic, Google, open-source models via Hugging Face).

• Excellent communication; ability to influence across engineering, product, security, and risk.

Nice to Have

• GPU systems & inference optimization (CUDA/NCCL, TensorRT-LLM, vLLM, TGI); Ray/Spark/Databricks for distributed training/inference.

• Observability: Prometheus/Grafana, OpenTelemetry, ML observability (e.g., WhyLabs, Arize), data quality (Great Expectations).

• Event streaming and real-time systems (Kafka/Event Hubs), micro-batching, CQRS.

• Search & knowledge systems (Elastic, OpenSearch, Knowledge Graphs).

Tech You'll Use (Illustrative)

• Cloud & Infra: Azure (AKS, Functions, App Service, Event Hubs, API Management, Key Vault, Private Link, Monitor), Terraform/Bicep, GitHub Actions/Azure DevOps.

• AI/ML: Python, PyTorch, ONNX, MLflow, Hugging Face, LangChain/LangGraph, OpenAI/Azure OpenAI, Anthropic, vector DBs (FAISS/Milvus/Pinecone/pgvector/Cosmos DB vectors).

• Serving & Ops: KServe/Triton, vLLM/TensorRT-LLM, Prometheus/Grafana, OpenTelemetry, Great Expectations, ArgoCD/GitOps, OPA/Azure Policy.

• Data & Orchestration: Spark/Databricks, Ray, Airflow/Prefect, Kafka/Event Hubs, Feast/feature store patterns.

How You'll Measure Success

• Reliability & Performance: SLOs met for AI services (latency, availability, quality); scalable throughput and GPU/infra efficiency.

• Security & Compliance: Zero critical findings; auditable lineage and model documentation; RAI controls consistently applied.

• Developer Velocity: Time-to-first model and time-to-production reduced via reusable components and golden paths.

• Business Impact: Clear ROI, adoption across lines of business, measurable customer/employee experience improvements.

• Technical Leadership: Mentorship, architectural influence, and uplift across teams; strong cross-functional partnerships.

Notes

• Additional responsibilities may be assigned based on your career growth ambitions and evolving enterprise needs.

• This role is individual contributor senior technical leadership (Principal), driving impact through architecture, code, and influence rather than direct line management.

Salary:

Pay Type:

The above represents BMO Financial Group's pay range and type.

Salaries will vary based on factors such as location, skills, experience, education, and qualifications for the role, and may include a commission structure. Salaries for part-time roles will be pro-rated based on number of hours regularly worked. For commission roles, the salary listed above represents BMO Financial Group's expected target for the first year in this position.

BMO Financial Group's total compensation package will vary based on the pay type of the position and may include performance-based incentives, discretionary bonuses, as well as other perks and rewards. BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans. To view more details of our benefits, please visit: 

About Us

At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world.

As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one – for yourself and our customers. We'll support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From in-depth training and coaching, to manager support and network-building opportunities, we'll help you gain valuable experience, and broaden your skillset.

To find out more visit us at

BMO is committed to an inclusive, equitable and accessible workplace. By learning from each other's differences, we gain strength through our people and our perspectives. Accommodations are available on request for candidates taking part in all aspects of the selection process. To request accommodation, please contact your recruiter.

Note to Recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO, directly or indirectly, will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid, written and fully executed agency agreement contract for service to submit resumes.