Senior Data Engineer

OverviewJoin to apply for the Senior Data Engineer - GCP role at ScotiabankRequisition ID: 234564Join a purpose driven winning team, committed to results, in an inclusive and high-performing culture. The Wealth Data engineering team within the Global Wealth Engineering (GWE) is the key team in meeting the operational data needs of the various stakeholders within Wealth Management. The Lead Data Engineer will play a key role in designing and implementing data solutions using Big Data, Hadoop (Cloudera) and Google Cloud, working closely with the enterprise data team and data architects, solution architects, business systems analysts and data engineers.Is this role right for you? In this role you will:ResponsibilitiesHelping development efforts in ingesting and transforming data from various sources. Working in the Big Data/ Hadoop environment, should be hands on in writing code, building scripts, writing specifications and responsible for end to end delivery of data in the Enterprise Data Lake environmentBuild distributed, reliable and scalable data pipelines to ingest and process data from multiple data sourcesDesigning, building, operationalizing the data platform using Google Cloud Platform (GCP) data services such as DataProc, Dataflow, CloudSQL, BigQuery, CloudSpanner in combination with third parties such as Spark, Apache Beam/ composer, DBT, Cloud PubSub, Confluent Kafka, Cloud storage Cloud Functions & GithubDesigning and implementing data ingestion patterns that will support batch, streaming and API interface on both the Ingress and Egress.Guide a team of data engineers and work hands on in developing framework and custom code using best practices that will meet the demanding performance requirementsTake a lead in designing and building production data pipelines from data ingestion to consumption using GCP services, Java, Python, Scala, BigQuery, DBT, SQL etc.Experience using Cloud Dataflow using Java/Python for deploying streaming jobs in GCP as well as batch jobs using text/JSON files and writing them to BigQueryBuilding and managing data pipelines with a deep understanding of workflow orchestration, task scheduling and dependency managementProvide end-to-end technical guidance and expertise on how to effectively use Google Cloud to build solutions; creatively applying cloud infrastructure and platform services to help solve business problems; and communicating these approaches to different business usersProvide guidance on Implementing application logging, notification, jobs monitoring and performance monitoringSkills2+ years of experience in data engineering, performance optimization for large OLTP applicationsKnowledge of Hadoop HDFS, Hive, Pig, Flume and Sqoop.Knowledge with the primary managed data services within GCP, including DataProc, Dataflow, BigQuery/DBT, Cloud Spanner, Cloud SQL, Cloud Pub/Sub etc.Knowledge with Google Cloud Platform Databases (SQL, Spanner, PostgreSQL)Working experience in HQLGood knowledge of the concepts of Hadoop.Knowledge working with relational/NoSQL databasesKnowledge with data streaming and technologies such as Kafka, Spark-streaming etc.Knowledge with Infrastructure as Code (IaC) practices and frameworks like TerraformKnowledge of Java microservices and Spring BootStrong architecture knowledge with experience in providing technical solutions for cloud infrastructure.Working knowledge of developing and scaling JAVA REST services, using frameworks such as SpringGood communication and problem-solving skills. Ability to effectively convey ideas to business and technical teamsNice-To-Have SkillsUnderstanding of Wealth business line and the various data domains required for building an end to end solutionWhat's in it for you?Diversity, Equity, Inclusion & Allyship - We strive to create an inclusive culture where every employee is empowered to reach their fullest potential, respected for who they are, and are embraced through bias-free practices and inclusive values across Scotiabank. We embrace diversity and provide opportunities for all employee to learn, grow & participate through our various Employee Resource Groups (ERGs) that span across diverse gender identities, ethnicity, race, age, ability & veterans.Accessibility and Workplace Accommodations - We value the unique skills and experiences each individual brings to the Bank, and are committed to creating and maintaining an inclusive and accessible environment for everyone. Scotiabank continues to locate, remove and prevent barriers so that we can build a diverse and inclusive environment while meeting accessibility requirements.Upskilling through online courses, cross-functional development opportunities, and tuition assistance.Competitive Rewards program including bonus, flexible vacation, personal, sick days and benefits will start on day one.Dynamic Ecosystem - Free tea & coffee, universal washrooms, and lots of space for team collaboration.Location(s): Canada : Ontario : TorontoScotiabank is a leading bank in the Americas. Guided by our purpose: "for every future", we help our customers, their families and their communities achieve success through a broad range of advice, products and services, including personal and commercial banking, wealth management and private banking, corporate and investment banking, and capital markets.At Scotiabank, we value the unique skills and experiences each individual brings to the Bank, and are committed to creating and maintaining an inclusive and accessible environment for everyone. If you require accommodation (including, but not limited to, an accessible interview site, alternate format documents, ASL Interpreter, or Assistive Technology) during the recruitment and selection process, please let our Recruitment team know. If you require technical assistance, please click here. Candidates must apply directly online to be considered for this role. We thank all applicants for their interest in a career at Scotiabank; however, only those candidates who are selected for an interview will be contacted.Seniority levelMid-Senior levelEmployment typeFull-timeJob functionInformation TechnologyIndustriesBanking #J-18808-Ljbffr