Senior GCP Data Engineer

Lead Data Engineer – Wealth Data Engineering We are seeking a Lead Data Engineer to join our Wealth Data Engineering team. This role is ideal for a technically strong and hands-on professional with deep expertise in data engineering, cloud-based data platforms, and large-scale data processing. You’ll play a key part in designing and implementing enterprise-grade data solutions that support strategic decision-making and operational excellence within Wealth Management. As a Lead Data Engineer, you’ll work closely with data architects, business analysts, and enterprise data teams to build scalable, secure, and high-performance data pipelines using Big Data, Hadoop (Cloudera), and Google Cloud Platform (GCP). You will drive both the technical vision and implementation, guiding a team of engineers while remaining actively involved in development and delivery. Key Responsibilities Data Architecture and Design Design, develop, and maintain scalable data architectures for ingestion, transformation, and storage across cloud and on-premises environments. Define and implement data ingestion frameworks to support batch, streaming, and API-driven integrations. Partner with data and solution architects to ensure technical alignment with enterprise data strategies and architecture standards. Data Pipeline Development Build and optimize distributed, reliable, and scalable data pipelines using GCP services such as DataProc , Dataflow , BigQuery , CloudSQL , CloudSpanner , and Pub/Sub . Develop data transformation workflows using Spark , Apache Beam , DBT , Confluent Kafka , and Cloud Functions . Implement and manage workflow orchestration, task scheduling, and dependency management for data processing pipelines. Technical Leadership and Best Practices Provide hands-on technical guidance to data engineers, ensuring adherence to development standards and best practices. Lead design and code reviews, enforcing quality, scalability, and performance optimization. Collaborate with cross-functional teams to deliver high-quality data solutions that meet business needs and timelines. Performance, Monitoring, and Optimization Monitor and maintain data pipelines to ensure reliability, performance, and cost efficiency. Implement automated logging, alerting, and performance dashboards for data workflows. Troubleshoot issues in production environments and optimize data processing workloads. Cloud and Infrastructure Leverage Infrastructure as Code (IaC) tools such as Terraform to automate deployment and management of cloud resources. Apply best practices for cloud data architecture, ensuring scalability, resiliency, and compliance. Integrate data solutions across multiple environments, including hybrid and multi-cloud setups. Collaboration and Strategy Work closely with business stakeholders to understand data needs and translate them into technical requirements. Contribute to the strategic direction of data engineering initiatives and identify opportunities for innovation and automation. Foster a culture of continuous learning and improvement within the data engineering team. Required Skills and Experience 2+ years of hands-on experience in data engineering , focusing on large-scale data processing and performance optimization. Strong expertise in Hadoop ecosystem components such as HDFS , Hive , Pig , Flume , and Sqoop . Practical experience with Google Cloud Platform (GCP) data services including DataProc , Dataflow , BigQuery , DBT , Cloud Spanner , Cloud SQL , and Pub/Sub . Proficiency in SQL , HQL , and experience working with both relational and NoSQL databases. Knowledge of streaming frameworks such as Kafka and Spark Streaming . Experience in Java microservices and REST API development using Spring Boot . Familiarity with Infrastructure as Code (IaC) tools like Terraform . Strong architectural design capabilities, with the ability to build scalable, maintainable data solutions in the cloud. Excellent communication, problem-solving, and leadership skills, with the ability to collaborate effectively across technical and business teams. Preferred Qualifications Understanding of Wealth Management data domains and related business processes. Experience designing end-to-end data solutions supporting financial or investment applications.