Data Engineering Expert for Cloud-Based Data Ecosystem

2 weeks ago

Vancouver, British Columbia, Canada State Street Full time

About the Role

We are seeking a highly skilled Data Engineering Expert to join our team at State Street. As a key member of our Global Technology Services (GTS) organization, you will play a critical role in shaping the future of our data ecosystem.

Job Summary

The successful candidate will be responsible for designing, developing, and maintaining scalable data pipelines using pyspark on Databricks, adhering to best practices and emphasizing software engineering principles. You will also implement and optimize stream processing workflows using Kafka for real-time data ingestion and processing, as well as leverage Databricks platform on AWS to build and manage data processing workflows and analytics.

Main Responsibilities

  • Design and develop scalable data pipelines using pyspark on Databricks, ensuring adherence to best practices and software engineering principles.
  • Implement and optimize stream processing workflows using Kafka for real-time data ingestion and processing.
  • Leverage Databricks platform on AWS to build and manage data processing workflows and analytics.
  • Maintain robust testing methodologies, including unit testing, integration testing, and end-to-end testing, utilizing Python packages such as pytest.
  • Contribute to the pyspark/Python ecosystem by creating reusable components, maintaining internal PyPI packages, and exploring other common Python packages.
  • Monitor data pipelines, identify and resolve issues, and ensure data integrity and quality.

Requirements

  • Master's degree in computer science or a related field.
  • Minimum 8-10 years of real-world Data Engineering experience working on large-scale data projects.
  • Strong proficiency in pySpark, Python, and shell scripting, with a focus on software engineering best practices and a deep understanding of development lifecycle.
  • Experience working with workflow management tools such as Airflow.
  • Experience with stream processing technologies, preferably Kafka.
  • Familiarity with Avro data serialization format and its usage in data engineering workflows.
  • Expertise in using Databricks platform on AWS for data processing and analytics.
  • Solid understanding of data warehousing concepts and experience with Delta Lake and Parquet files.
  • Proficiency in SQL and experience with relational databases.
  • Strong testing skills, with experience in implementing and executing unit tests, integration tests, and end-to-end tests using Python packages such as pytest.
  • Familiarity with the Python ecosystem, including PyPI packages and their integration into data engineering workflows.

Salary Range

$100,000 - $160,000 per annum

About State Street

State Street is one of the largest custodian banks, asset managers, and asset intelligence companies in the world. We provide investment servicing, data & analytics, investment research & trading, and investment management to institutional clients. Our mission is to help our clients safeguard and steward the investments of millions of people, while delivering innovative solutions that meet their evolving needs.

  • Data Insights Specialist

    1 month ago

    Vancouver, British Columbia, Canada Data Privacy Full time

    Job Title: Data AnalystThe Data Analyst team is responsible for monitoring and investigating Bing Ads performance to quickly identify and resolve anomalies. Their goal is to connect seemingly unrelated data to maintain the health of the marketplace ecosystem.Key Responsibilities:Drive technical analysis to improve monitoring tools and processesCoordinate...

  • Data Engineer

    1 month ago

    Vancouver, British Columbia, Canada Electronic Arts Inc. Full time

    Job Title: Sr. Data DeveloperWe are seeking an experienced Sr. Data Developer to join our team at Electronic Arts Inc. The ideal candidate will have a strong background in data engineering, with expertise in designing and implementing scalable data pipelines, data modeling, and data warehousing.Key Responsibilities:Design and implement efficient data...

  • Senior Data Insights Specialist

    2 weeks ago

    Vancouver, British Columbia, Canada Data Privacy Full time

    Job OverviewThe Data Analyst role at Data Privacy is a key position focused on monitoring and investigating Bing Ads performance to quickly identify and resolve anomalies. The team aims to connect seemingly unrelated data to maintain the health of the marketplace ecosystem.Key ResponsibilitiesDrive technical analysis to improve monitoring tools and resolve...

  • Key Performance Data Insights Specialist

    2 weeks ago

    Vancouver, British Columbia, Canada Data Privacy Full time

    The Monitoring and Investigation Team at Microsoft is focused on analyzing Bing Ads performance to quickly identify and resolve anomalies. The team's primary goal is to maintain the health of the marketplace ecosystem by connecting seemingly unrelated data.Job DescriptionThis role involves driving technical analysis, improving monitoring tools, and...

  • Senior Cloud Data Engineer

    4 weeks ago

    Vancouver, British Columbia, Canada Providence Health Care Full time

    Job SummaryProvidence Health Care is seeking a highly skilled Senior Cloud Data Engineer to join our Data Analytics team. As a key member of our team, you will be responsible for designing, developing, and maintaining our data infrastructure, including databases, warehouses, and pipelines. Your expertise in cloud-based data engineering and analytics will...

  • Data Engineering Leader

    1 month ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the RoleAs a Data Engineering Leader at Amazon Data Services Canada, Inc., you will be responsible for mentoring and leading a team of engineers to develop and maintain data systems that support our customers' sustainability initiatives. You will work closely with our Product and SDE teams to build data systems that simplify bringing new models to...

  • Data Engineering Leader

    4 weeks ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the Role:We are seeking a highly skilled Data Engineering Leader to join our team at Amazon Data Services Canada, Inc. As a Data Engineering Leader, you will be responsible for mentoring and leading engineers, guiding the team towards best data development practices, and defining the priorities and vision of the group.Key Responsibilities:Lead the...

  • Data Management Expert

    1 week ago

    Vancouver, British Columbia, Canada KPMG-Canada Full time

    OverviewKPMG in Canada is a leading advisory firm that helps organizations unlock the value of their data. As a Data Management Expert, you will be part of our Technology Consulting practice, which provides strategic solutions to clients around the world.Salary and BenefitsThe estimated salary for this position is $98,000 to $127,000 per year, depending on...

  • Data Engineer

    4 weeks ago

    Vancouver, British Columbia, Canada Later Full time

    Later is seeking a highly skilled Data Engineer to join its team and contribute to the development of its cloud-based data infrastructure. As a Data Engineer, you will be responsible for designing and implementing data pipelines that support machine learning and business intelligence across the organization.This is an excellent opportunity to work with a...

  • Sustainability Data Architect

    3 days ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    Amazon Data Services Canada, Inc. is a leader in cloud computing and artificial intelligence.About UsWe are committed to furthering our culture of inclusion, with ten employee-led affinity groups and innovative benefit offerings. Our team puts a high value on work-life balance, offering flexibility in working hours and encouraging you to find your own...

  • Data Ecosystems Product Visionary

    2 weeks ago

    Vancouver, British Columbia, Canada CLIO Full time

    Transformative Role at CLIOClio, a global leader in revolutionizing the legal experience, seeks a visionary Staff Product Manager to spearhead the Data Ecosystem team. As a key player, you will drive the strategic planning, roadmap execution, and vision for deeply technical solutions that unify data across Clio's diverse product suite.About This...

  • Data Ecosystem Unification Lead

    1 week ago

    Vancouver, British Columbia, Canada CLIO Full time

    Clio is more than just a tech company, we are a global leader that is transforming the legal experience for all by innovating and simplifying complex processes.What You'll Work On:As the Data Ecosystem Unification Lead at Clio, you will lead the vision, strategic planning, and roadmap execution of deeply technical solutions that unify data across Clio's...

  • Senior Cloud Data Engineer

    4 weeks ago

    Vancouver, British Columbia, Canada Providence Health Care Full time

    Job SummaryProvidence Health Care is seeking a highly skilled Senior Cloud Data Engineer to join our Data Analytics team. As a key member of our team, you will be responsible for designing, developing, and maintaining our data infrastructure, including databases, warehouses, and pipelines. Your expertise in cloud data engineering will enable us to provide a...

  • Data Engineer

    1 month ago

    Vancouver, British Columbia, Canada Amazon Development Centre Canada ULC Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Amazon Development Centre Canada ULC. As a key member of our Marketplace team, you will be responsible for designing and implementing scalable data architectures to support our cloud-based software platform.Key ResponsibilitiesDesign and develop data pipelines to integrate...

  • Data Engineering Leader for Sustainability Metrics

    1 week ago

    Vancouver, British Columbia, Canada Amazon Data Services Canada, Inc. Full time

    About the JobAs a Data Engineer Leader with Amazon Data Services Canada, Inc., you will be responsible for leading a team of engineers in building data systems that simplify bringing new models to customers and enhance visibility into our underlying modeling process. You will mentor and guide your team towards best practices in data engineering, while...

  • Cloud Data Architect Manager

    1 month ago

    Vancouver, British Columbia, Canada Providence Health Care Full time

    Job SummaryProvidence Health Care is seeking a highly skilled Cloud Data Architect to lead our team in designing, developing, and implementing secure and scalable cloud-based data infrastructure. This role will be responsible for developing and implementing cloud-based data solutions that address operational and clinical challenges within the...

  • Data Engineering Lead for Cloud Services

    2 weeks ago

    Vancouver, British Columbia, Canada Microsoft Canada Full time

    About Microsoft CanadaMicrosoft Canada is a hub for passionate innovators to collaborate, envision what can be, and take their careers further. This is a world of endless possibilities, driven by innovation, openness, and limitless thinking in a cloud-enabled environment.The Azure Data engineering team at Microsoft Canada is pioneering the transformation of...

  • Data Management Lead

    1 month ago

    Vancouver, British Columbia, Canada NTT DATA, Inc. Full time

    Drive Innovation and Growth with NTT DATANTT DATA is a global leader in business and technology services, serving 75% of the Fortune Global 100. We're committed to helping clients innovate, optimize, and transform for long-term success.About the RoleWe're seeking an experienced Application Portfolio Management Lead to join our team in Vancouver, British...

  • Senior Data Engineer

    4 weeks ago

    Vancouver, British Columbia, Canada Amazon Development Centre Canada ULC Full time

    About the RoleAmazon Development Centre Canada ULC is seeking a highly skilled Sr. Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data systems and architectures.Key ResponsibilitiesDesign and implement data pipelines and architectures to support...

  • Data Engineering Expert

    1 week ago

    Vancouver, British Columbia, Canada Electronic Arts Full time

    About the RoleWe're looking for a skilled Data Engineer II to join our team at Electronic Arts. This role involves working on various data-related projects, from designing and implementing efficient data pipelines to collaborating with analysts to develop technical specifications.Key Responsibilities:Develop and maintain large-scale data systems, ensuring...