Senior Data Engineer, Databricks on AWS, AVP

2 weeks ago

Vancouver, British Columbia, Canada State Street Full time

About the Role

We are seeking a highly skilled Data Engineer to lead our cloud data platforms and drive the transition to a modern data ecosystem. As a key member of our team, you will design, implement, and optimize data pipelines using pyspark on Databricks, adhering to best practices and emphasizing software engineering principles.

Key Responsibilities

  • Design and develop scalable data pipelines using pyspark on Databricks, ensuring data integrity and quality.
  • Implement and optimize stream processing workflows using Kafka for real-time data ingestion and processing.
  • Utilize Parquet and Avro-formatted data files for efficient storage and retrieval, ensuring data schema compatibility and evolution.
  • Collaborate closely with data analysts and scientists to understand their requirements and provide reliable and timely data solutions.
  • Implement robust testing methodologies, including unit testing, integration testing, and end-to-end testing, utilizing Python packages such as pytest.
  • Contribute to the pyspark/Python ecosystem by creating reusable components, maintaining internal PyPI packages, and exploring other common Python packages.
  • Monitor data pipelines, identify and resolve issues, and ensure data integrity and quality.

Requirements

  • Master's degree in computer science or a related field.
  • Minimum 8-10 years of real-world Data Engineering experience working on large-scale data projects.
  • Strong proficiency in pySpark, Python, and shell scripting, with a focus on software engineering best practices and a deep understanding of development lifecycle.
  • Experience working with workflow management tools such as Airflow.
  • Experience with stream processing technologies, preferably Kafka.
  • Familiarity with Avro data serialization format and its usage in data engineering workflows.
  • Expertise in using Databricks platform on AWS for data processing and analytics.
  • Solid understanding of data warehousing concepts and experience with Delta Lake and Parquet files.
  • Proficiency in SQL and experience with relational databases.
  • Strong testing skills, with experience in implementing and executing unit tests, integration tests, and end-to-end tests using Python packages such as pytest.
  • Familiarity with the Python ecosystem, including PyPI packages and their integration into data engineering workflows.

About State Street

State Street is one of the largest custodian banks, asset managers, and asset intelligence companies in the world. We provide investment servicing, data & analytics, investment research & trading, and investment management to institutional clients. We are committed to creating a great work environment and offer competitive benefits packages, flexible work programs, and a wealth of development programs and educational support to help you reach your full potential.

  • Senior Data Engineer, Databricks on AWS, AVP

    3 weeks ago

    Vancouver, British Columbia, Canada State Street Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at State Street. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines using pyspark on Databricks, adhering to best practices and emphasizing software engineering principles.Key ResponsibilitiesDesign and develop data...

  • Senior Data Engineer, Databricks on AWS, AVP

    3 weeks ago

    Vancouver, British Columbia, Canada State Street Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at State Street. As a Data Engineer, you will play a key role in shaping the future of our data ecosystem by designing, implementing, and optimizing data pipelines using modern data management techniques.Key ResponsibilitiesDesign and develop scalable data pipelines using pyspark on...

  • Senior Data Engineer, Databricks on AWS, Technical Lead

    2 months ago

    Vancouver, British Columbia, Canada State Street Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at State Street. As a key member of our Global Technology Services (GTS) organization, you will play a critical role in shaping the future of our data ecosystem.Key ResponsibilitiesDesign, develop, and maintain scalable data pipelines using pyspark on Databricks, adhering to...

  • Senior Data Engineer, Databricks on AWS, Technical Lead

    2 months ago

    Vancouver, British Columbia, Canada State Street Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at State Street. As a key member of our Global Technology Services (GTS) organization, you will play a critical role in shaping the future of our data ecosystem.Key ResponsibilitiesDesign, develop, and maintain scalable data pipelines using pyspark on Databricks, adhering to...

  • Senior Data Engineer

    4 weeks ago

    Vancouver, British Columbia, Canada GeoComply Full time

    About GeoComplyWe're a leading provider of geolocation compliance solutions, trusted by global brands and regulators for over 10 years. Our innovative technologies help ensure regulatory compliance, combat online fraud, and protect businesses from financial losses.The RoleWe're seeking a Senior Data Engineer to join our team, contributing to the construction...

  • Senior Data Engineer

    4 weeks ago

    Vancouver, British Columbia, Canada GeoComply Full time

    About GeoComplyWe're a leading provider of geolocation compliance solutions, trusted by global brands and regulators for over 10 years. Our innovative technologies help ensure regulatory compliance, combat online fraud, and protect businesses from financial losses.The RoleWe're seeking a Senior Data Engineer to join our team, contributing to the construction...

  • Senior Data Engineer

    3 weeks ago

    Vancouver, British Columbia, Canada GeoComply Full time

    About GeoComplyWe are a leading provider of geolocation compliance solutions, trusted by customers worldwide. Our innovative technologies help ensure regulatory compliance, combat online fraud, and protect businesses from financial losses.The RoleThe Senior Data Engineer will play a critical role in constructing data pipelines that transform raw data into...

  • Senior Data Engineer

    1 month ago

    Vancouver, British Columbia, Canada GeoComply Full time

    About GeoComplyWe are a leading provider of geolocation compliance solutions, trusted by global brands and regulators for over 10 years. Our innovative technologies help ensure regulatory compliance, combat bad online actors, and protect businesses from fraud.The RoleThe Senior Data Engineer will contribute to our success by designing and implementing data...

  • Senior Data Engineer

    1 month ago

    Vancouver, British Columbia, Canada GeoComply Full time

    About GeoComplyWe are a leading provider of geolocation compliance solutions, trusted by global brands and regulators for over 10 years. Our innovative technologies help ensure regulatory compliance, combat bad online actors, and protect businesses from fraud.The RoleThe Senior Data Engineer will contribute to our success by designing and implementing data...

  • Data Engineer III

    3 weeks ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a key member of our team, you will be responsible for mentoring and leading engineers, guiding them towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesCollaborate with...

  • Data Engineer III

    3 weeks ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a key member of our team, you will be responsible for mentoring and leading engineers, guiding them towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesCollaborate with...

  • Data Engineer III

    4 weeks ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a key member of our team, you will be responsible for mentoring and leading engineers, guiding them towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesCollaborate with...

  • Data Engineer III

    4 weeks ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a key member of our team, you will be responsible for mentoring and leading engineers, guiding them towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesCollaborate with...

  • Senior Data Scientist

    3 weeks ago

    Vancouver, British Columbia, Canada Electronic Arts Full time

    Job SummaryWe are seeking a highly skilled Senior Data Scientist to join our team at Electronic Arts. As a key member of our data science team, you will be responsible for designing and implementing large-scale data systems, building end-to-end machine learning pipelines, and deploying models to cloud infrastructure.ResponsibilitiesData Management: Design...

  • Senior Data Scientist

    2 weeks ago

    Vancouver, British Columbia, Canada Electronic Arts Full time

    About the RoleWe are seeking a highly skilled Senior Data Scientist to join our team at Electronic Arts. As a key member of our engineering team, you will be responsible for designing and implementing large-scale data systems, building end-to-end machine learning pipelines, and deploying services to cloud infrastructure.ResponsibilitiesData Management:...

  • Data Engineer III

    4 weeks ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a key member of our team, you will be responsible for mentoring and leading engineers, guiding them towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesWork with our...

  • Data Engineer III

    4 weeks ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a key member of our team, you will be responsible for mentoring and leading engineers, guiding them towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesWork with our...

  • Data Engineering Leader

    3 weeks ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the RoleAs a Data Engineering Leader at Amazon Data Services Canada, Inc., you will be responsible for mentoring and leading engineers in the development of data systems that simplify bringing new models to customers and enhance the visibility into our underlying modeling process.Our team operates the public Customer Carbon Footprint Tool and we have a...

  • Data Engineer

    3 weeks ago

    Vancouver, British Columbia, Canada Amazon Full time

    About the RoleWe are seeking an experienced Data Engineer to join our AWS Kumo team. As a Data Engineer, you will play a critical role in designing, building, and maintaining our data architecture, ensuring high-quality data is available to support business decisions.Key ResponsibilitiesDevelop and improve our current data architecture, data quality,...

  • Data Engineering Leader

    3 weeks ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the RoleWe are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a Data Engineering Leader, you will be responsible for mentoring and leading engineers, guiding the team towards best data development practices, and defining the priorities and vision of the group.Key ResponsibilitiesWork with our...