Senior Data Engineer, Databricks on AWS Transformation Specialist

2 weeks ago

Vancouver, British Columbia, Canada State Street Full time
Job Overview

We are seeking a skilled Senior Data Engineer to join our team in Boston. As a key member of our Global Technology Services (GTS) group, you will play a crucial role in driving the company's digital transformation and expanding business capabilities using industry best practices and advanced technologies.

Responsibilities
  • Design, develop, and maintain scalable data pipelines using pyspark on Databricks, adhering to best practices and emphasizing software engineering principles.
  • Implement and optimize stream processing workflows using Kafka for real-time data ingestion and processing.
  • Utilize Parquet and Avro-formatted data files for efficient storage and retrieval, ensuring data schema compatibility and evolution.
  • Leverage Databricks platform on AWS to build and manage data processing workflows and analytics, while adhering to development lifecycle standards.
  • Harness the power of Databricks Delta Lake and Parquet files for data warehousing, query optimization, and data versioning.
  • Collaborate closely with data analysts and scientists to understand their requirements and provide reliable and timely data solutions.
  • Implement robust testing methodologies, including unit testing, integration testing, and end-to-end testing, utilizing Python packages such as pytest.
  • Contribute to the pyspark/Python ecosystem by creating reusable components, maintaining internal PyPI packages, and exploring other common Python packages.
  • Monitor data pipelines, identify and resolve issues, and ensure data integrity and quality.
Requirements
  • Master's degree in computer science or a related field.
  • Minimum 8-10 years of real-world Data Engineering experience working on large-scale data projects.
  • Strong proficiency in pySpark, Python, and shell scripting, with a focus on software engineering best practices and a deep understanding of development lifecycle.
  • Experience working with workflow management tools such as Airflow.
  • Experience with stream processing technologies, preferably Kafka.
  • Familiarity with Avro data serialization format and its usage in data engineering workflows.
  • Expertise in using Databricks platform on AWS for data processing and analytics.
  • Solid understanding of data warehousing concepts and experience with Delta Lake and Parquet files.
  • Proficiency in SQL and experience with relational databases.
  • Strong testing skills, with experience in implementing and executing unit tests, integration tests, and end-to-end tests using Python packages such as pytest.
  • Familiarity with the Python ecosystem, including PyPI packages and their integration into data engineering workflows.
What We Offer

We offer a collaborative environment where technology skills and innovation are valued in a global organization. We're looking for top technical talent to join our team and deliver creative technology solutions that help us become an end-to-end, next-generation financial services company.

Benefits
  • Competitive salary: $120,000 - $150,000 per year.
  • Ongoing training and professional development opportunities.
  • A collaborative and dynamic work environment.
  • A comprehensive benefits package, including medical, dental, and vision insurance, as well as 401(k) matching.

  • Senior Data Engineer, Databricks on AWS, AVP

    2 weeks ago

    Vancouver, British Columbia, Canada State Street Full time

    Job Overview:State Street is seeking a highly skilled Senior Data Engineer to join our Global Technology Services team. As a key member of our data engineering team, you will play a crucial role in shaping the future of our data ecosystem.About the Role:We are looking for an experienced Data Engineer with a strong background in modern data management and...

  • Senior Data Engineer, Databricks on AWS, Technical Lead

    3 weeks ago

    Vancouver, British Columbia, Canada State Street Full time

    Role OverviewWe are seeking an experienced Data Engineer to join our team at State Street. As a Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data pipelines using Databricks and AWS technologies. This is an exciting opportunity to work with cutting-edge data engineering tools and contribute to the development...

  • Senior Data Engineering Specialist

    2 weeks ago

    Vancouver, British Columbia, Canada Amazon Full time

    Unlock your potential as a Senior Data Engineering Specialist at Amazon.About the RoleWe are seeking an experienced data engineer to join our team in designing, managing, and continuously enhancing our data and analytics infrastructure. This includes process automation/improvements, developing automated reporting solutions/tools, and improving the ability of...

  • Senior Data Engineer

    1 month ago

    Vancouver, British Columbia, Canada GeoComply Full time

    About GeoComplyWe are a leading provider of geolocation compliance solutions, trusted by customers worldwide. Our innovative technologies help ensure regulatory compliance, combat online fraud, and protect businesses from financial losses.The RoleThe Senior Data Engineer will play a critical role in constructing data pipelines that transform raw data into...

  • Data Engineering Expert for Cloud-Based Data Ecosystem

    2 weeks ago

    Vancouver, British Columbia, Canada State Street Full time

    About the RoleWe are seeking a highly skilled Data Engineering Expert to join our team at State Street. As a key member of our Global Technology Services (GTS) organization, you will play a critical role in shaping the future of our data ecosystem.Job SummaryThe successful candidate will be responsible for designing, developing, and maintaining scalable data...

  • Data Engineer

    1 month ago

    Vancouver, British Columbia, Canada Vancity Full time

    About the Role:We are seeking a highly skilled Data Engineer to join our team at Vancity. As a Data Engineer, you will be responsible for designing and building efficient and reliable data pipelines, cleansing and normalizing diverse datasets for Analytics.Key Responsibilities:Design and build data pipelines using big data and analytics programming languages...

  • Data Engineer

    2 weeks ago

    Vancouver, British Columbia, Canada GeoComply Full time

    About the RoleWe are seeking a Senior Data Engineer to join our team at GeoComply, a leading provider of geolocation compliance solutions. As a Senior Data Engineer, you will play a crucial role in constructing data pipelines that transform raw data into actionable business insights.The ideal candidate will have relevant experience in database technologies,...

  • Senior Data Architect

    4 weeks ago

    Vancouver, British Columbia, Canada Vancouver Coastal Health Full time

    Key ResponsibilitiesVancouver Coastal Health is seeking a skilled Principal Data Engineer to join the Data & Analytics Team. The successful candidate will be responsible for leading technical teams in the design, development, and deployment of cloud solutions to support assigned projects.Key responsibilities include:Leading the planning and execution of...

  • Data Engineer

    7 hours ago

    Vancouver, British Columbia, British Columbia, Canada Intelliswift Software Full time

    Job Title: Data EngineerLocation: Mexico (100% Remote)Duration: 6+ Months contract.Must have: Python, ADF, Databricks, Scala and KafkaGood to have: Spring Boot and UI DevelopmentKey Responsibilities:Provide production support for data pipelines and applications using Python, ADF, Azure Databricks, and Kafka.Monitor, troubleshoot, and resolve issues in a...

  • Data Engineer

    1 month ago

    Vancouver, British Columbia, Canada Amazon Full time

    About the RoleWe are seeking an experienced Data Engineer to join our AWS Kumo team. As a Data Engineer, you will play a critical role in designing, building, and maintaining our data architecture, ensuring high-quality data is available to support business decisions.Key ResponsibilitiesDevelop and improve our current data architecture, data quality,...

  • Senior Data Engineering Specialist

    2 weeks ago

    Vancouver, British Columbia, Canada Demonware Full time

    Job OverviewDemonware, a leading developer of online and data services for the gaming industry, is seeking an experienced Senior Data Engineering Specialist. As a key member of our talented data engineering team, you will play a crucial role in managing our large-scale data footprint.

  • Data Engineering Leader

    1 month ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the Role:We are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a Data Engineering Leader, you will be responsible for mentoring and leading engineers, guiding the team towards best data development practices, and defining the priorities and vision of the group.Key Responsibilities:Lead the...

  • Data Engineering Leader for Sustainability Metrics

    2 weeks ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the JobAs a Data Engineer Leader with Amazon Data Services Canada, Inc., you will be responsible for leading a team of engineers in building data systems that simplify bringing new models to customers and enhance visibility into our underlying modeling process. You will mentor and guide your team towards best practices in data engineering, while...

  • Data Engineer, AWS Kumo Analytics

    1 month ago

    Vancouver, British Columbia, Canada Amazon Full time

    Job Summary: We are looking for a Data Engineer with a passion for data and insights who can develop and improve the current data architecture, collaborate with Data Scientists, and partner with Business Analysts to build and verify hypothesis to improve the AWS Support business.Develop and improve the current data architecture, data quality, monitoring and...

  • Data Engineer

    3 weeks ago

    Vancouver, British Columbia, Canada Ledcor Full time

    Job SummaryLedcor is seeking a Data Engineer to join our team on a full-time basis. As a Data Engineer, you will be responsible for building, maintaining, and optimizing our data infrastructure to empower informed decision-making. Your role will involve developing, testing, and refining data from diverse source systems using modern data lakehouse...

  • Data Engineer II

    2 weeks ago

    Vancouver, British Columbia, Canada Electronic Arts Full time

    Job OverviewWe are seeking a skilled Data Engineer II - Senior Data Specialist to join our team at Electronic Arts. This is an exciting opportunity for a talented data professional to leverage their expertise in designing and implementing efficient, scalable, and reliable data pipelines.

  • Senior Data Engineer for Competitive Intelligence

    2 weeks ago

    Vancouver, British Columbia, Canada Tbwa ChiatDay Inc Full time

    We are seeking a highly skilled Senior Data Engineer to join our fast-paced Engineering team at Jungle Scout.About the RoleThis is a unique opportunity to work with a talented team of data scientists and engineers to develop cutting-edge Amazon competitive intelligence tools. As a Senior Data Engineer, you will be responsible for designing, building, and...

  • Senior Software Development Engineer, AWS IAM Specialist

    4 weeks ago

    Vancouver, British Columbia, Canada Amazon Full time

    Senior Software Development Engineer, AWS IAMAmazon Web Services Identity and Access Management (AWS IAM) is a critical component of the AWS cloud, responsible for identity management, authentication, and authorization. Our mission is to provide customers with state-of-the-art controls to manage access across all their AWS environments. We are seeking a...

  • Senior Data Science Expert

    4 weeks ago

    Vancouver, British Columbia, Canada Microsoft Full time

    At Microsoft, we're committed to empowering every person and organization on the planet to achieve more. Our Real-Time Intelligence team is leading the transformation of real-time analytics in the world of data, and we're looking for a Senior Data Scientist to join our journey.The team provides comprehensive solutions and a robust platform that enables users...

  • Senior Machine Learning Engineer for Embedded Finance

    3 weeks ago

    Vancouver, British Columbia, Canada Marqeta Full time

    OverviewAs a Senior Machine Learning Engineer at Marqeta, you will be part of a team dedicated to building AI/ML solutions for optimizing the security and efficiency of the embedded finance ecosystem.ResponsibilitiesDevelop and maintain machine learning infrastructure to enable data scientists to develop, train, evaluate, deploy, and operate ML models and...