Senior Data Engineer, Databricks on AWS, Technical Lead
2 months ago
We are seeking a highly skilled Senior Data Engineer to join our team at State Street. As a key member of our Global Technology Services (GTS) organization, you will play a critical role in shaping the future of our data ecosystem.
Key Responsibilities- Design, develop, and maintain scalable data pipelines using pyspark on Databricks, adhering to best practices and emphasizing software engineering principles.
- Implement and optimize stream processing workflows using Kafka for real-time data ingestion and processing.
- Utilize Parquet and Avro-formatted data files for efficient storage and retrieval, ensuring data schema compatibility and evolution.
- Leverage Databricks platform on AWS to build and manage data processing workflows and analytics, while adhering to development lifecycle standards.
- Harness the power of Databricks Delta Lake and Parquet files for data warehousing, query optimization, and data versioning.
- Collaborate closely with data analysts and scientists to understand their requirements and provide reliable and timely data solutions.
- Implement robust testing methodologies, including unit testing, integration testing, and end-to-end testing, utilizing Python packages such as pytest.
- Contribute to the pyspark/Python ecosystem by creating reusable components, maintaining internal PyPI packages, and exploring other common Python packages.
- Monitor data pipelines, identify and resolve issues, and ensure data integrity and quality.
- Stay up-to-date with the latest trends and technologies in data engineering, software development, and testing practices, and actively share knowledge with the team.
- Master's degree in computer science or a related field.
- Minimum 8-10 years of real-world Data Engineering experience working on large-scale data projects.
- Strong proficiency in pySpark, Python, and shell scripting, with a focus on software engineering best practices and a deep understanding of development lifecycle.
- Experience working with workflow management tools such as Airflow.
- Experience with stream processing technologies, preferably Kafka.
- Familiarity with Avro data serialization format and its usage in data engineering workflows.
- Expertise in using Databricks platform on AWS for data processing and analytics.
- Solid understanding of data warehousing concepts and experience with Delta Lake and Parquet files.
- Proficiency in SQL and experience with relational databases.
- Strong testing skills, with experience in implementing and executing unit tests, integration tests, and end-to-end tests using Python packages such as pytest.
- Familiarity with the Python ecosystem, including PyPI packages and their integration into data engineering workflows.
- Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment.
- Strong communication skills and ability to effectively communicate complex technical concepts to non-technical stakeholders.
- Experience working with reporting tools such as Tableau.
- Past experience working on Machine Learning projects.
- Past experience working in finance.
State Street is one of the largest custodian banks, asset managers, and asset intelligence companies in the world. From technology to product innovation, we're making our mark on the financial services industry. For more than two centuries, we've been helping our clients safeguard and steward the investments of millions of people. We provide investment servicing, data & analytics, investment research & trading, and investment management to institutional clients.
We truly believe our employees' diverse backgrounds, experiences, and perspectives are a powerful contributor to creating an inclusive environment where everyone can thrive and reach their maximum potential while adding value to both our organization and our clients. We warmly welcome candidates of diverse origin, background, ability, age, sexual orientation, gender identity, and personality.
We offer a collaborative environment where technology skills and innovation are valued in a global organization. We're looking for top technical talent to join our team and deliver creative technology solutions that help us become an end-to-end, next-generation financial services company.
-
Senior Data Engineer, Databricks on AWS, AVP
3 weeks ago
Vancouver, British Columbia, Canada State Street Full timeAbout the RoleWe are seeking a highly skilled Data Engineer to join our team at State Street. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines using pyspark on Databricks, adhering to best practices and emphasizing software engineering principles.Key ResponsibilitiesDesign and develop data...
-
Senior Data Engineer, Databricks on AWS, AVP
2 weeks ago
Vancouver, British Columbia, Canada State Street Full timeAbout the RoleWe are seeking a highly skilled Data Engineer to join our team at State Street. As a Data Engineer, you will play a key role in shaping the future of our data ecosystem by designing, implementing, and optimizing data pipelines using modern data management techniques.Key ResponsibilitiesDesign and develop scalable data pipelines using pyspark on...
-
Senior Data Engineer, Databricks on AWS, AVP
2 weeks ago
Vancouver, British Columbia, Canada State Street Full timeAbout the RoleWe are seeking a highly skilled Data Engineer to lead our cloud data platforms and drive the transition to a modern data ecosystem. As a key member of our team, you will design, implement, and optimize data pipelines using pyspark on Databricks, adhering to best practices and emphasizing software engineering principles.Key...
-
Senior Big Data Engineer
1 month ago
Vancouver, British Columbia, Canada Data Engineer Jobs Full timeSenior Big Data EngineerWe are seeking a highly skilled Senior Big Data Engineer to join our WW Promotions team at Amazon. As a key member of our team, you will be responsible for designing, processing, and managing large datasets across various data platforms. Your expertise will enable us to identify the best set of products to promote, select the most...
-
Senior Big Data Engineer
1 month ago
Vancouver, British Columbia, Canada Data Engineer Jobs Full timeSenior Big Data EngineerWe are seeking a highly skilled Senior Big Data Engineer to join our WW Promotions team at Amazon. As a key member of our team, you will be responsible for designing, processing, and managing large datasets across various data platforms. Your expertise will enable us to identify the best set of products to promote, select the most...
-
Amazon Senior Big Data Engineer
4 weeks ago
Vancouver, British Columbia, Canada Data Engineer Jobs Full timeSenior Big Data EngineerWe are seeking a highly skilled Senior Big Data Engineer to join our team at Amazon. As a key member of our data engineering team, you will be responsible for designing, processing, and managing large datasets across various data platforms. Your expertise will enable us to identify the best set of products to promote, select the most...
-
Amazon Senior Big Data Engineer
4 weeks ago
Vancouver, British Columbia, Canada Data Engineer Jobs Full timeSenior Big Data EngineerWe are seeking a highly skilled Senior Big Data Engineer to join our team at Amazon. As a key member of our data engineering team, you will be responsible for designing, processing, and managing large datasets across various data platforms. Your expertise will enable us to identify the best set of products to promote, select the most...
-
Amazon Senior Big Data Engineer
1 month ago
Vancouver, British Columbia, Canada Data Engineer Jobs Full timeSenior Big Data EngineerWe are seeking a highly skilled Senior Big Data Engineer to join our team at Amazon. As a key member of our WW Promotions team, you will play a critical role in designing, processing, and managing large datasets to drive business growth and profitability.Key Responsibilities:Design and implement scalable data pipelines to ingest and...
-
Amazon Senior Big Data Engineer
1 month ago
Vancouver, British Columbia, Canada Data Engineer Jobs Full timeSenior Big Data EngineerWe are seeking a highly skilled Senior Big Data Engineer to join our team at Amazon. As a key member of our WW Promotions team, you will play a critical role in designing, processing, and managing large datasets to drive business growth and profitability.Key Responsibilities:Design and implement scalable data pipelines to ingest and...
-
Senior Data Engineer
2 months ago
Vancouver, British Columbia, Canada Data Engineer Jobs Full timeAbout the RoleWe are seeking a highly skilled Senior Data Engineer to join our Promotions team at Amazon. As a key member of our team, you will be responsible for designing, processing, and managing large datasets across various data platforms.Key ResponsibilitiesDesign and implement data pipelines to ingest and enable analytics at scale.Develop and maintain...
-
Senior Data Engineer
2 months ago
Vancouver, British Columbia, Canada Data Engineer Jobs Full timeAbout the RoleWe are seeking a highly skilled Senior Data Engineer to join our Promotions team at Amazon. As a key member of our team, you will be responsible for designing, processing, and managing large datasets across various data platforms.Key ResponsibilitiesDesign and implement data pipelines to ingest and enable analytics at scale.Develop and maintain...
-
Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full timeAbout the RoleWe are seeking a highly skilled Software Development Manager to lead the delivery of customer-facing sustainability products within our AWS Infrastructure Sustainability Tech organization. This team builds solutions to integrate sustainability data across AWS infrastructure and operations, supporting AWS customers' sustainability goals and our...
-
Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full timeAbout the RoleWe are seeking a highly skilled Software Development Manager to lead the delivery of customer-facing sustainability products within our AWS Infrastructure Sustainability Tech organization. This team builds solutions to integrate sustainability data across AWS infrastructure and operations, supporting AWS customers' sustainability goals and our...
-
Senior Data Engineer
4 weeks ago
Vancouver, British Columbia, Canada GeoComply Full timeAbout GeoComplyWe're a leading provider of geolocation compliance solutions, trusted by global brands and regulators for over 10 years. Our innovative technologies help ensure regulatory compliance, combat online fraud, and protect businesses from financial losses.The RoleWe're seeking a Senior Data Engineer to join our team, contributing to the construction...
-
Senior Data Engineer
4 weeks ago
Vancouver, British Columbia, Canada GeoComply Full timeAbout GeoComplyWe're a leading provider of geolocation compliance solutions, trusted by global brands and regulators for over 10 years. Our innovative technologies help ensure regulatory compliance, combat online fraud, and protect businesses from financial losses.The RoleWe're seeking a Senior Data Engineer to join our team, contributing to the construction...
-
Data Engineer III
3 weeks ago
Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full timeAbout the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a key member of our team, you will be responsible for mentoring and leading engineers, guiding them towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesCollaborate with...
-
Data Engineer III
3 weeks ago
Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full timeAbout the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a key member of our team, you will be responsible for mentoring and leading engineers, guiding them towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesCollaborate with...
-
Data Engineer III
4 weeks ago
Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full timeAbout the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a key member of our team, you will be responsible for mentoring and leading engineers, guiding them towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesCollaborate with...
-
Data Engineer III
4 weeks ago
Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full timeAbout the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a key member of our team, you will be responsible for mentoring and leading engineers, guiding them towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesCollaborate with...
-
Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full timeAbout the RoleWe are seeking a highly skilled Software Development Manager to lead our Sustainability Tech team at Amazon Web Services (AWS). As a key member of our organization, you will be responsible for delivering high-impact sustainability products and features that integrate sustainability data across AWS infrastructure and operations.Key...
