Senior Data Engineer

OverviewRequisition ID: 234564Join a purpose driven winning team, committed to results, in an inclusive and high-performing culture.The Wealth Data engineering team within the Global Wealth Engineering (GWE) is the key team in meeting the operational data needs of the various stake holders within Wealth Management. The Lead Data Engineer will play a key role in designing and implementing data solutions using Big Data, Hadoop (Cloudera) and Google cloud working closely with the enterprise data team and data architects, solution architects, business systems analyst and data engineers.ResponsibilitiesIngest and transform data from various sources in a Big Data/Hadoop environment; write code, build scripts, write specifications, and be responsible for end-to-end delivery of data in the Enterprise Data Lake environment.Build distributed, reliable and scalable data pipelines to ingest and process data from multiple data sources.Design, build and operationalize the data platform using Google Cloud Platform (GCP) data services such as DataProc, Dataflow, CloudSQL, BigQuery, Cloud Spanner, in combination with third parties such as Spark, Apache Beam/Composer, DBT, Cloud Pub/Sub, Confluent Kafka, Cloud Storage, Cloud Functions and GitHub.Design and implement data ingestion patterns that support batch, streaming and API interfaces on both ingress and egress.Guide a team of data engineers and contribute hands-on to develop frameworks and custom code following best practices to meet performance requirements.Lead in designing and building production data pipelines from ingestion to consumption using GCP services, Java, Python, Scala, BigQuery, DBT, SQL, etc.Experience using Cloud Dataflow with Java/Python for deploying streaming jobs in GCP as well as batch jobs and writing results to BigQuery.Build and manage data pipelines with a deep understanding of workflow orchestration, task scheduling and dependency management.Provide end-to-end technical guidance on effectively using Google Cloud to build solutions; apply cloud infrastructure and platform services to solve business problems and communicate approaches to business users.Provide guidance on implementing application logging, notifications, job monitoring and performance monitoring.Qualifications2+ years of experience in data engineering, with performance optimization for large OLTP applications.Knowledge of Hadoop HDFS, Hive, Pig, Flume and Sqoop.Knowledge of primary managed data services within GCP, including DataProc, Dataflow, BigQuery/DBT, Cloud Spanner, Cloud SQL, Cloud Pub/Sub, etc.Knowledge of Google Cloud Platform databases (SQL, Spanner, PostgreSQL).Working experience in HQL.Good knowledge of Hadoop concepts.Experience with relational/NoSQL databases.Knowledge of data streaming technologies such as Kafka and Spark Streaming.Knowledge of Infrastructure as Code (IaC) practices and frameworks like Terraform.Knowledge of Java microservices and Spring Boot.Strong architecture knowledge with experience in providing technical solutions for cloud infrastructure.Experience developing and scaling Java REST services using Spring.Good communication and problem-solving skills; ability to convey ideas to both business and technical teams.Nice-To-Have SkillsUnderstanding of Wealth business line and data domains required for end-to-end solutions.What's in it for you?Diversity, Equity, Inclusion & Allyship — we foster an inclusive culture and provide opportunities for learning, growth and participation through Employee Resource Groups (ERGs).Accessibility and Workplace Accommodations — commitment to an inclusive and accessible environment; accommodation requests during recruitment are welcomed.Upskilling through online courses, cross-functional development opportunities, and tuition assistance.Competitive rewards program including bonus, flexible vacation, personal and sick days and benefits from day one.Dynamic ecosystem — amenities and space for team collaboration.Location(s): Canada : Ontario : TorontoScotiabank is a leading bank in the Americas. Guided by our purpose: for every future, we help customers and communities achieve success through a broad range of services including wealth management and private banking.If you require accommodation during the recruitment and selection process, please let our Recruitment team know. Candidates must apply directly online to be considered for this role. We thank all applicants for their interest in a career at Scotiabank; however, only those candidates selected for an interview will be contacted. #J-18808-Ljbffr