Lead Data Engineer

3 weeks ago

Toronto, Canada VDart Full time

Role: Lead Data Engineer -GCP

Location: Toronto, ON

Full Time


Job Description for Lead Data Engineer (GCP)

The Google Lead Data Engineer will play a key role in designing and implementing the Data Hub using Google cloud components working closely with the enterprise data team and data architects, solution architects, business systems analyst and data engineers.


Key accountabilities/ Experience

• Designing, building, operationalizing the Wealth Data Hub (WDH) using Google Cloud Platform (GCP) data services such as DataProc, Dataflow, CloudSQL, BigQuery, CloudSpanner in combination with third parties such as Spark, Apache Beam/ composer, DBT, Cloud PubSub, Confluent Kafka, Cloud storage Cloud Functions & Github

• Designing and implementing data ingestion patterns that will support batch, streaming and API interface on both the Ingress and Egress.

• Guide a team of data engineers and work hands on in developing framework and custom code using best practices that will meet the demanding performance requirements

• Take a lead in designing and building production data pipelines from data ingestion to consumption using GCP services, Java, Python, Scala, BigQuery, DBT, SQL etc.

• Experience using Cloud Dataflow using Java/Python for deploying streaming jobs in GCP as well as batch jobs using text/JSON files and writing them to BigQuery

• Building and managing data pipelines with a deep understanding of workflow orchestration, task scheduling and dependency management

• Ability to do proof of technology using GCP technologies and work with data architects, solution architects to achieve the desired results and performance.

• Provide end-to-end technical guidance and expertise on how to effectively use Google Cloud to build solutions; creatively applying cloud infrastructure and platform services to help solve business problems; and communicating these approaches to different business users

• Provide guidance on Implementing application logging, notification, jobs monitoring and performance monitoring

Functional Competencies and Skills

• 10+ years of experience in data engineering, performance optimization for large OLTP applications with a minimum of 4+ years of working experience as Google Cloud Platform (GCP) developer

• Experience with the primary managed data services within GCP, including DataProc, Dataflow, BigQuery/DBT, Cloud Spanner, Cloud SQL, Cloud Pub/Sub etc.

• Experience with Google Cloud Platform Databases (SQL, Spanner, PostgreSQL)

• Experience working with relational/NoSQL databases

• Experience with data streaming and technologies such as Kafka, Spark-streaming etc.

• Experience with Infrastructure as Code (IaC) practices and frameworks like Terraform

• Knowledge of Java microservices and Spring Boot

• Strong architecture knowledge with experience in providing technical solutions for cloud infrastructure.

• Working knowledge of developing and scaling JAVA REST services, using frameworks such as Spring

• Good communication and problem-solving skills. Ability to effectively convey ideas to business and technical teams

  • Lead Data Engineer

    4 days ago

    Toronto, Canada Insight Global Full time

    Must-haves - 10-12 years experience working within data engineering roles - 5-7 years experience with AWS cloud services including: EMR, MWAA, LakeFormation, Glue, DMS, QuickSight, and AWS Athena - 8+ years experience with big data technologies including: Apache Spark, Apache Hudi, Hive, Airflow, Trino, and Databricks - CICD - Terraform for sure ,...

  • Lead Data Engineer

    6 days ago

    Toronto, Canada Insight Global Full time

    Must-haves - 10-12 years experience working within data engineering roles - 5-7 years experience with AWS cloud services including: EMR, MWAA, LakeFormation, Glue, DMS, QuickSight, and AWS Athena - 8+ years experience with big data technologies including: Apache Spark, Apache Hudi, Hive, Airflow, Trino, and Databricks - CICD - Terraform for sure ,...

  • Lead Data Engineer

    5 days ago

    Toronto, Canada Insight Global Full time

    Must-haves - 10-12 years experience working within data engineering roles - 5-7 years experience with AWS cloud services including: EMR, MWAA, LakeFormation, Glue, DMS, QuickSight, and AWS Athena - 8+ years experience with big data technologies including: Apache Spark, Apache Hudi, Hive, Airflow, Trino, and Databricks - CICD - Terraform for sure ,...

  • Lead Data Engineer

    4 days ago

    Toronto, Canada Insight Global Full time

    Must-haves - 10-12 years experience working within data engineering roles - 5-7 years experience with AWS cloud services including: EMR, MWAA, LakeFormation, Glue, DMS, QuickSight, and AWS Athena - 8+ years experience with big data technologies including: Apache Spark, Apache Hudi, Hive, Airflow, Trino, and Databricks - CICD - Terraform for sure ,...

  • Manager/Lead- Data Engineering

    3 weeks ago

    Toronto, Canada hireVouch Full time

    Our client is a Toronto-based GCP consulting company that is focused on helping a variety of businesses through cloud-based solutions. They are growing their team and are looking for a passionate Data Engineering Manager/Lead to continue to guide them to success. In this role, you will be responsible for leading a team of data engineers in designing,...

  • Manager/Lead- Data Engineering

    3 weeks ago

    Toronto, Canada hireVouch Full time

    Our client is a Toronto-based GCP consulting company that is focused on helping a variety of businesses through cloud-based solutions. They are growing their team and are looking for a passionate Data Engineering Manager/Lead to continue to guide them to success. In this role, you will be responsible for leading a team of data engineers in designing,...

  • Manager/Lead- Data Engineering

    3 weeks ago

    Toronto, Canada hireVouch Full time

    Our client is a Toronto-based GCP consulting company that is focused on helping a variety of businesses through cloud-based solutions. They are growing their team and are looking for a passionate Data Engineering Manager/Lead to continue to guide them to success. In this role, you will be responsible for leading a team of data engineers in designing,...

  • Manager/Lead- Data Engineering

    3 weeks ago

    Toronto, Canada hireVouch Full time

    Our client is a Toronto-based GCP consulting company that is focused on helping a variety of businesses through cloud-based solutions. They are growing their team and are looking for a passionate Data Engineering Manager/Lead to continue to guide them to success.In this role, you will be responsible for leading a team of data engineers in designing,...

  • Data Engineering Practice Lead

    4 weeks ago

    Toronto, Canada McMillan Shakespeare Limited Full time

    The McMillan Shakespeare Group (MMS) is a trusted provider of salary packaging, novated leasing, disability plan management and support co-ordination, asset management and related financial products and services. From our origins in 1988 when we created Australia’s salary packaging industry to today, MMS has a proud history of innovation and exceptional...

  • Sr/Lead Data Engineer

    4 weeks ago

    Toronto, Canada Asset Management Industry Full time

    Good day Data Engineers ! Insight Global is looking for 6 Sr or Lead Data Engineers to join a large Asset Management company in Toronto on a full time basis. The successful candidates will be responsible for leading projects end-to-end within the Data Distribution Team or Asset Management Team . T

  • Sr/Lead Data Engineer

    4 weeks ago

    Toronto, Canada Asset Management Industry Full time

    Good day Data Engineers ! Insight Global is looking for 6 Sr or Lead Data Engineers to join a large Asset Management company in Toronto on a full time basis. The successful candidates will be responsible for leading projects end-to-end within the Data Distribution Team or Asset Management Team . This role is heavily hands on and requires strong...

  • Sr/Lead Data Engineer

    3 weeks ago

    Toronto, Canada Asset Management Industry Full time

    Good day Data Engineers ! Insight Global is looking for 6 Sr or Lead Data Engineers to join a large Asset Management company in Toronto on a full time basis. The successful candidates will be responsible for leading projects end-to-end within the Data Distribution Team or Asset Management Team . This role is heavily hands on and requires strong...

  • Sr/Lead Data Engineer

    4 weeks ago

    Toronto, Canada Asset Management Industry Full time

    Good day Data Engineers!Insight Global is looking for 6 Sr or Lead Data Engineers to join a large Asset Management company in Toronto on a full time basis. The successful candidates will be responsible for leading projects end-to-end within the Data Distribution Team or Asset Management Team. This role is heavily hands on and requires strong experience with...

  • Sr/Lead Data Engineer

    3 weeks ago

    Toronto, Canada Asset Management Industry Full time

    Good day Data Engineers ! Insight Global is looking for 6 Sr or Lead Data Engineers to join a large Asset Management company in Toronto on a full time basis. The successful candidates will be responsible for leading projects end-to-end within the Data Distribution Team or Asset Management Team . This role is heavily hands on and requires strong...

  • Sr/Lead Data Engineer

    4 weeks ago

    Toronto, Canada Asset Management Industry Full time

    Good day Data Engineers!Insight Global is looking for 6 Sr or Lead Data Engineers to join a large Asset Management company in Toronto on a full time basis. The successful candidates will be responsible for leading projects end-to-end within the Data Distribution Team or Asset Management Team. This role is heavily hands on and requires strong experience with...

  • Sr/Lead Data Engineer

    3 weeks ago

    Toronto, Canada Asset Management Industry Full time

    Good dayData Engineers !Insight Global is looking for 6 Sr or LeadData Engineersto join a large Asset Management company in Toronto on a full time basis. The successful candidates will be responsible for leading projects end-to-end within theData Distribution TeamorAsset Management Team . This role is heavily hands on and requires strong experience with...

  • Sr/Lead Data Engineer

    4 weeks ago

    Toronto, Canada Asset Management Industry Full time

    Good day Data Engineers!Insight Global is looking for 6 Sr or Lead Data Engineers to join a large Asset Management company in Toronto on a full time basis. The successful candidates will be responsible for leading projects end-to-end within the Data Distribution Team or Asset Management Team. This role is heavily hands on and requires strong experience with...

  • Sr/Lead Data Engineer

    4 weeks ago

    Toronto, Canada Asset Management Industry Full time

    Good day Data Engineers ! Insight Global is looking for 6 Sr or Lead Data Engineers to join a large Asset Management company in Toronto on a full time basis. The successful candidates will be responsible for leading projects end-to-end within the Data Distribution Team or Asset Management Team . This role is heavily hands on and requires strong experience...

  • Sr/Lead Data Engineer

    4 weeks ago

    Toronto, Canada Asset Management Industry Full time

    Good day Data Engineers!Insight Global is looking for 6 Sr or Lead Data Engineers to join a large Asset Management company in Toronto on a full time basis. The successful candidates will be responsible for leading projects end-to-end within the Data Distribution Team or Asset Management Team. This role is heavily hands on and requires strong experience with...

  • Lead Data Engineer

    2 weeks ago

    Toronto, Canada Crypto Recruiters Full time

    We are actively searching for a Lead Data Engineer to join our team on a permanent basis. This role does require onsite work as you will be leading a team that is building a data product that will give insights to investors in the crypto and Web3 industry. In this role, you will be responsible for designing and building cutting-edge data analysis systems,...