Lead Data Engineer

2 weeks ago

Toronto, Ontario, Canada Princeton IT Services Full time

Job Title: Lead Data Engineer – Python, PySpark & SQL

Location: Canada

Job Type: Full time contract

We are looking for a strong Lead Data Engineer with deep experience in Python, PySpark, SQL, and AWS to design, develop, and optimize large-scale data pipelines. This role requires strong hands-on coding skills, the ability to validate and process complex raw data, and expertise in running and tuning PySpark jobs on EMR.

Responsibilities

  • Build scalable data ingestion and transformation pipelines using Python, PySpark, and SQL.
  • Process raw CSV/text files from AWS S3, including validating headers, schema checks, and malformed file detection.
  • Convert raw data into structured DataFrames and implement reusable data quality checks.
  • Develop advanced transformations using SQL/PySpark (Window functions, LAG(), grouping logic, date gap detection, etc.).
  • Deploy and tune PySpark applications on AWS EMR, optimizing executor memory, cores, shuffle behavior, and cluster performance.
  • Work with AWS services such as S3, EMR, Glue, Lambda, IAM.
  • Debug performance issues (OOM errors, shuffle spill, GC problems) and improve pipeline reliability.
  • Lead design discussions, code reviews, and mentor junior engineers.

Required Skills

  • 8+ years of experience in Data Engineering.
  • Expert Python (file processing, scripting, validation automation).
  • Strong PySpark (DataFrames, job tuning, distributed processing).
  • Advanced SQL (analytical functions, performance tuning).
  • Hands-on with AWS data stack: S3, EMR, Glue, Lambda.
  • Strong understanding of Spark memory allocation, YARN container usage, and EMR resource tuning.
  • Excellent debugging, communication, and problem-solving skills.

Nice to Have

  • Airflow or Databricks experience.
  • Terraform or CloudFormation.
  • Experience with data lake formats (Delta, Iceberg, Hudi).

Job Type: Full-time

Pay: $50.00-$53.00 per hour

Experience:

  • Data Engineer: 10 years (required)
  • Pyspark: 4 years (required)
  • Python: 6 years (required)
  • AWS: 4 years (required)

  • Data Engineer Lead

    1 week ago

    Toronto, Ontario, Canada Themesoft Inc. Full time

    Location : Toronto - 4 days a week.Role: Data Engineer LeadSkills:- Data Engineer Lead - Relational DB and No SQL, Hadoop, Spark, Kafka, ETL, Data Modelling

  • Lead Data Engineer

    7 days ago

    Toronto, Ontario, Canada Mastercard Full time

    Our PurposeMastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we're helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships...

  • Lead Data Engineer

    2 days ago

    Toronto, Ontario, Canada Scotiabank Full time

    Requisition ID: 237458Join a purpose driven winning team, committed to results, in an inclusive and high-performing culture.The Wealth Data engineering team within the Global Wealth Engineering (GWE) is the key team in meeting the operational data needs of the various stake holders within Wealth Management. The Lead Data Engineer will play a key role in...

  • Data Engineering Lead

    24 hours ago

    Toronto, Ontario, Canada Fitch Group Full time

    Data Engineering LeadAbout the TeamJoin the Data Marketplace Tech team within the CSO organization—a highly visible group responsible for designing, building, and scaling solutions that free up data and empower business decisions across Fitch. The team collaborates closely with stakeholders across the organization to deliver next-generation data...

  • Lead Azure Data Engineer

    2 weeks ago

    Toronto, Ontario, Canada Insight Global Full time

    JOB DESCRIPTIONInsight Global is seeking a Lead Data Engineer (Azure Specialist) to lead cloud-based data initiatives and deliver robust solutions for enterprise-scale projects for one of Canada's largest banks. This role focuses on Azure technologies and data engineering to ensure optimal performance and scalability.Azure Expertise: Design and implement...

  • Lead Databricks Data Engineer

    2 weeks ago

    Toronto, Ontario, Canada CloudTech Innovations Full time

    Lead Data Engineer – DatabricksLocation:Onsite – Toronto, CanadaType:ContractAbout the RoleWe are looking for a Lead Data Engineer with 8–10 years of experience who can quickly assess complex data problems, make sound technical decisions, and drive solutions end to end. This role requires deep hands-on experience with Databricks and modern lakehouse...

  • Lead Data Platform Engineer

    2 weeks ago

    Toronto, Ontario, Canada OCS Ontario Cannabis Store Full time

    About UsThe Ontario Cannabis Store provides safe, responsible access to recreational cannabis for adults 19 and older. We operate the sole legal online store for recreational cannabis in Ontario and are the provincial wholesaler of cannabis for private retail stores.Working at the OCS is a unique opportunity to be part of an agile start-up in a...

  • Lead Data Platform Engineer

    13 hours ago

    Toronto, Ontario, Canada OCS Ontario Cannabis Store Full time

    About UsThe Ontario Cannabis Store provides safe, responsible access to recreational cannabis for adults 19 and older. We operate the sole legal online store for recreational cannabis in Ontario and are the provincial wholesaler of cannabis for private retail stores.Working at the OCS is a unique opportunity to be part of an agile start-up in a...

  • Lead Data Engineer

    2 days ago

    Toronto, Ontario, Canada RBC Full time

    Job DescriptionWhat is the opportunity?This is a Senior Technical Lead, Data Engineering position which is part of fast growingWealth Management Technology & Solution (WMTS)Data Service team which will work with multiple RBC teams, upstream/downstream system consumers and services providers, 3rd party vendor partners, and operation partners. WMTS Data...

  • Forward Deployed Engineers

    4 days ago

    Toronto, Ontario, Canada Data Intellect Full time

    Company Description At Data Intellect it has never been just about data or technology, they are our tools. It's about human intellect, collaboration and providing solutions for the most complex of challenges.We do this by living the [DI] code:We are Problem Solvers who are Humble, possess a Can-do Attitude with a focus on Togetherness."We are not big on...