Lead Data Engineer

2 weeks ago

Toronto, Ontario, Canada Princeton IT Services Full time

Job Title: Lead Data Engineer – Python, PySpark & SQL

Location: Canada

Job Type: Full time contract

We are looking for a strong Lead Data Engineer with deep experience in Python, PySpark, SQL, and AWS to design, develop, and optimize large-scale data pipelines. This role requires strong hands-on coding skills, the ability to validate and process complex raw data, and expertise in running and tuning PySpark jobs on EMR.

Responsibilities

  • Build scalable data ingestion and transformation pipelines using Python, PySpark, and SQL.
  • Process raw CSV/text files from AWS S3, including validating headers, schema checks, and malformed file detection.
  • Convert raw data into structured DataFrames and implement reusable data quality checks.
  • Develop advanced transformations using SQL/PySpark (Window functions, LAG(), grouping logic, date gap detection, etc.).
  • Deploy and tune PySpark applications on AWS EMR, optimizing executor memory, cores, shuffle behavior, and cluster performance.
  • Work with AWS services such as S3, EMR, Glue, Lambda, IAM.
  • Debug performance issues (OOM errors, shuffle spill, GC problems) and improve pipeline reliability.
  • Lead design discussions, code reviews, and mentor junior engineers.

Required Skills

  • 8+ years of experience in Data Engineering.
  • Expert Python (file processing, scripting, validation automation).
  • Strong PySpark (DataFrames, job tuning, distributed processing).
  • Advanced SQL (analytical functions, performance tuning).
  • Hands-on with AWS data stack: S3, EMR, Glue, Lambda.
  • Strong understanding of Spark memory allocation, YARN container usage, and EMR resource tuning.
  • Excellent debugging, communication, and problem-solving skills.

Nice to Have

  • Airflow or Databricks experience.
  • Terraform or CloudFormation.
  • Experience with data lake formats (Delta, Iceberg, Hudi).

Job Type: Full-time

Pay: $50.00-$53.00 per hour

Experience:

  • Data Engineer: 10 years (required)
  • Pyspark: 4 years (required)
  • Python: 6 years (required)
  • AWS: 4 years (required)

  • Principal Data Engineer

    1 day ago

    Toronto, Ontario, Canada JamLabs Data Science Full time

    About the CompanyWhy JamLabs? At JamLabs, we're not just another data consultancy. We're a Toronto-based firm founded by leading engineers and data scientists, with deep roots in capital markets and cloud engineering. Our mission is to transform data into strategic assets for financial services firms, driving growth and innovation through tailored...

  • Lead Data Engineer

    1 week ago

    Toronto, Ontario, Canada Mastercard Full time

    Our PurposeMastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we're helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships...

  • Lead Data Engineer

    6 days ago

    Toronto, Ontario, Canada RBC Full time

    Job DescriptionWhat is the opportunity?At RBC, our data engineering team enhances visibility into assets across the Public Cloud and Application Security landscape. Our mission is to provide clear insights into digital infrastructure, enabling effective identification and management of security risks. We harness industry-leading tools like Databricks,...

  • Lead Data Engineer

    4 days ago

    Toronto, Ontario, Canada Scotiabank Full time

    Requisition ID: 237458Join a purpose driven winning team, committed to results, in an inclusive and high-performing culture.The Wealth Data engineering team within the Global Wealth Engineering (GWE) is the key team in meeting the operational data needs of the various stake holders within Wealth Management. The Lead Data Engineer will play a key role in...

  • Data Engineering Lead

    3 days ago

    Toronto, Ontario, Canada Fitch Group Full time

    Data Engineering LeadAbout the TeamJoin the Data Marketplace Tech team within the CSO organization—a highly visible group responsible for designing, building, and scaling solutions that free up data and empower business decisions across Fitch. The team collaborates closely with stakeholders across the organization to deliver next-generation data...

  • Data Engineering Lead

    1 week ago

    Toronto, Ontario, Canada myGwork - LGBTQ+ Business Community Full time

    This job is with Fitch Group, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly.Data Engineering LeadAbout the TeamJoin the Data Marketplace Tech team within the CSO organization—a highly visible group responsible for designing, building, and scaling...

  • Azure Data Engineer Lead

    17 hours ago

    Toronto, Ontario, Canada MethodHub Full time

    Job Title: Lead Data EngineerLocation: Canada (Hybrid Toronto)12 + Months Contract PositionAbout the Role:We are seeking an experienced Lead Data Engineer to spearhead the design, development, and implementation of our enterprise data platform. This role will be instrumental in shaping and driving our data engineering roadmap, ensuring scalability,...

  • Lead Azure Data Engineer

    2 weeks ago

    Toronto, Ontario, Canada Insight Global Full time

    JOB DESCRIPTIONInsight Global is seeking a Lead Data Engineer (Azure Specialist) to lead cloud-based data initiatives and deliver robust solutions for enterprise-scale projects for one of Canada's largest banks. This role focuses on Azure technologies and data engineering to ensure optimal performance and scalability.Azure Expertise: Design and implement...

  • Lead Databricks Data Engineer

    2 weeks ago

    Toronto, Ontario, Canada CloudTech Innovations Full time

    Lead Data Engineer – DatabricksLocation:Onsite – Toronto, CanadaType:ContractAbout the RoleWe are looking for a Lead Data Engineer with 8–10 years of experience who can quickly assess complex data problems, make sound technical decisions, and drive solutions end to end. This role requires deep hands-on experience with Databricks and modern lakehouse...

  • Lead Data Platform Engineer

    2 weeks ago

    Toronto, Ontario, Canada OCS Ontario Cannabis Store Full time

    About UsThe Ontario Cannabis Store provides safe, responsible access to recreational cannabis for adults 19 and older. We operate the sole legal online store for recreational cannabis in Ontario and are the provincial wholesaler of cannabis for private retail stores.Working at the OCS is a unique opportunity to be part of an agile start-up in a...