Senior Data Pipeline Engineer

1 month ago


Vancouver, Canada Celestial Systems Full time

Celestial Systems, a name synonymous with Enterprise Softwaredevelopment since 2001. By designingreliable software,delivering quality services, and developing valuable partnerships, we are on a journey of continuousinnovation. Ourenterprise-level experience in engineering enables us to offer critical development services fororganizations to besuccessful in the digital ages. Moreover, at Celestial, we focus on adhering to the complete softwaredevelopmentlifecycle process. Our experts are curious to work with the latest technologies and are extremelypassionate aboutsoftware development. Whether its Frontend Development, Backend, DevOps, QA and Testing or CloudHosting, ourtailor-made technology solutions, provide answers to the problems.We are ISO certified enterprise-grade software company.

Roles and Responsibilities

Celestial is rapidly expanding or Enterprise Data Services team to offer our customers a single point for all requirements including, Data Engineering, Data Analytics, Data Protection and Data Ethics/Governance.

We are looking for Senior Data Pipeline Engineers (multiple positions) for ongoing projects, within the Data Services and ML/AI Group. If you are motivated and willing to work in a startup mode on exciting NextGen projects do read on.

Requirements

• 4+ years of experience in big data engineering, with a focus on Spark and SQL.

• 4+ Years Experience in developing Large-scale ETL Pipelines using AWS services not limited to IAM, Glue, Lambda, S3, Athena, SNS, SQS, DynamoDB, RDS, EMR, ECS, Route 53 and Redshift.

• Highly skilled in Python, PostgreSQL/RDS/Redshift, and writing complex SQL queries and analysis of data correlations.

• Design, construct, and manage Data Lake environments including Data ingestion, staging, Data quality monitoring, and business modeling, Data Schema Normalization, etc.

• Develop, construct, secure, test and maintain architectures (databases and large-scale processing systems)

• Design, develop and implement technical solutions in SQL environment (databases, SQL, data marts, integrations)

• Experience in tools like Jenkins, GIT and/or AWS CodePipeline/CodeBuild/CodeDeploy

• Experience building data-driven unit test suites for data platforms and modeling highly dimensional datasets

• Strong functional understanding of Sagemaker, Jupyter and other Data Science tools on AWS

PREFERRED SKILLS

Experience designing and building big data architectures using tools such as Hadoop, Hive, or Kafka.

ADDITIONAL NOTES

If you are applying for Part-time/Contract role please email your resume with contact details, and "Hourly Consulting Rate Expectations".

Apply for this Job

Drop your Resume *


Attach or Drop Your resume
#J-18808-Ljbffr

  • Vancouver, British Columbia, Canada Kabam Full time

    At Kabam, we're constantly raising the bar of excellence in free-to-play gaming. We hire for passion and diversity so that we can triumph in our collective skill. Our teams are made up of a fusion of personalities and interests bound together by a collaborative nature, fun-loving culture, and a drive to do what it takes to make great games. And this is where...


  • Vancouver, Canada Fraser Health Full time

    Salary range The salary range for this position is CAD $45.46 - $65.35 / hour Why Fraser Health? Fraser Health is responsible for the delivery of hospital and community-based health services to over 1.9 million people in 20 diverse communities from Burnaby to Fraser Canyon on the traditional territories of the Coast Salish and Nlaka'pamux Nations....


  • Vancouver, Canada Amplitude Full time

    Amplitude is a leading digital analytics platform that helps companies unlock the power of their products. More than 2,300 customers, including Atlassian, Jersey Mike’s, NBCUniversal, Shopify, and Under Armour, rely on Amplitude to gain self-service visibility into the entire customer journey. Amplitude guides companies every step of the way as they...

  • Senior Engineer, Data

    5 hours ago


    North Vancouver, BC, Canada Stellar Recruitment Inc. Full time

    Are you passionate about leveraging cutting-edge technologies to drive innovation? As a Senior DevOps Engineer with expertise in Machine Learning (ML), you'll spearhead the development of robust pipelines for ML systems in production, while championing best practices in MLOps. Responsibilities:Guide the development of pipelines for ML systems in production,...


  • Vancouver, Canada Zortechsolutions Full time

    Role: Data Engineer-SQL, Python, ETLLocation: Vancouver/CANADADuration: Full TimeJob DescriptionKey Skills required: We are looking for Senior Data Engineering Candidates based out of the Canada. Expertise in SQL, Python and ETL flows is a must have. Medium expertise/working knowledge in Scala is also needed, as our data pipelines are written in both Scala...


  • Vancouver, Canada Zortechsolutions Full time

    Role: Data Engineer-SQL, Python, ETLLocation: Vancouver/CANADADuration: Full TimeJob DescriptionKey Skills required: We are looking for Senior Data Engineering Candidates based out of the Canada. Expertise in SQL, Python and ETL flows is a must have. Medium expertise/working knowledge in Scala is also needed, as our data pipelines are written in both Scala...


  • Vancouver, Canada Zortechsolutions Full time

    Role: Data Engineer-SQL, Python, ETLLocation: Vancouver/CANADADuration: Full TimeJob DescriptionKey Skills required: We are looking for Senior Data Engineering Candidates based out of the Canada. Expertise in SQL, Python and ETL flows is a must have. Medium expertise/working knowledge in Scala is also needed, as our data pipelines are written in both Scala...

  • Coming Soon

    12 hours ago


    Vancouver, Canada ActiveState Full time

    Senior Data Engineer  ActiveState is securing open source software supply chains and enhancing dependency management with an on-demand SaaS Platform and CLI tool that lets DevSecOps automatically build any runtime environment using any open source language ecosystem on any platform. We’re building an ambitious language distribution platform so that no...

  • Coming Soon

    11 hours ago


    Vancouver, Canada ActiveState Full time

    Senior Data Engineer  ActiveState is securing open source software supply chains and enhancing dependency management with an on-demand SaaS Platform and CLI tool that lets DevSecOps automatically build any runtime environment using any open source language ecosystem on any platform. We’re building an ambitious language distribution platform so that no...

  • Lead Engineer

    2 weeks ago


    Vancouver, Canada Zonda Full time

    Job Title: Lead Engineer (Data) Location: Vancouver, BC / GVRD Reports To: Director of Data Engineering Job Summary: We are seeking a highly skilled and motivated individual to join our team as a Lead Engineer in Data Operation. As a Lead Engineer in Data Operations, you will be responsible for driving the development of modern data pipelines and...

  • Lead Engineer

    2 weeks ago


    Vancouver, Canada Zonda Full time

    Job Title: Lead Engineer (Data) Location: Vancouver, BC / GVRD Reports To: Director of Data Engineering Job Summary: We are seeking a highly skilled and motivated individual to join our team as a Lead Engineer in Data Operation. As a Lead Engineer in Data Operations, you will be responsible for driving the development of modern data pipelines and...

  • Lead Engineer

    2 weeks ago


    Vancouver, Canada Zonda Full time

    Job Title: Lead Engineer (Data) Location: Vancouver, BC / GVRD Reports To: Director of Data Engineering Job Summary: We are seeking a highly skilled and motivated individual to join our team as a Lead Engineer in Data Operation. As a Lead Engineer in Data Operations, you will be responsible for driving the development of modern data pipelines and...


  • Vancouver, Canada Vancouver Airport Authority Full time

    YVR is made up of a team of diverse people who are working collaboratively to Connect BC Proudly to the World. Safety is at the core of everything we do; we’re innovative, fun, and we invest in our people. We’re a BC Top Employer for 14 years standing, with high engagement scores, an abundance of learning and development opportunities, and a holistic...


  • Vancouver, Canada Vancouver Airport Authority Full time

    YVR is made up of a team of diverse people who are working collaboratively to Connect BC Proudly to the World. Safety is at the core of everything we do; we’re innovative, fun, and we invest in our people. We’re a BC Top Employer for 14 years standing, with high engagement scores, an abundance of learning and development opportunities, and a holistic...


  • Vancouver, Canada Vancouver Airport Authority Full time

    YVR is made up of a team of diverse people who are working collaboratively to Connect BC Proudly to the World. Safety is at the core of everything we do; we’re innovative, fun, and we invest in our people. We’re a BC Top Employer for 14 years standing, with high engagement scores, an abundance of learning and development opportunities, and a holistic...


  • Vancouver, Canada Industrial Light & Magic Full time

    Job Summary: This role is considered Hybrid, which means the employee will work 2-3 days onsite at a Company designated location and occasionally from home. Position Summary A Pipeline TD at Industrial Light & Magic provides direct support to productions and artists. They provide general troubleshooting of pipeline tools in a fast-paced, feature film visual...


  • Vancouver, Canada Industrial Light & Magic Full time

    Job Summary: This role is considered Hybrid, which means the employee will work 2-3 days onsite at a Company designated location and occasionally from home. Position Summary A Pipeline TD at Industrial Light & Magic provides direct support to productions and artists. They provide general troubleshooting of pipeline tools in a fast-paced, feature film visual...


  • Vancouver, Canada Industrial Light & Magic Full time

    Job Summary: This role is considered Hybrid, which means the employee will work 2-3 days onsite at a Company designated location and occasionally from home. Position Summary A Pipeline TD at Industrial Light & Magic provides direct support to productions and artists. They provide general troubleshooting of pipeline tools in a fast-paced, feature film visual...

  • Lead Engineer

    2 weeks ago


    Vancouver, Canada Zonda Full time

    Job Title:Lead Engineer (Data) Location:Vancouver, BC / GVRD Reports To:Director of Data EngineeringJob Summary: We are seeking a highly skilled and motivated individual to join our team as a Lead Engineer in Data Operation. As a Lead Engineer in Data Operations, you will be responsible for driving the development of modern data pipelines and solutions that...

  • Senior MLOps Engineer

    12 hours ago


    North Vancouver, Canada Stellar Recruitment Inc. Full time

    Are you passionate about leveraging cutting-edge technologies to drive innovation? Do you thrive in dynamic environments where collaboration and continuous improvement are the norm? As a Senior DevOps Engineer with expertise in Machine Learning (ML), you'll spearhead the development of robust pipelines for ML systems in production, while championing best...