Senior Data Pipeline Engineer

3 weeks ago


Vancouver, Canada Celestial Systems Full time

Celestial Systems, a name synonymous with Enterprise Softwaredevelopment since 2001. By designingreliable software,delivering quality services, and developing valuable partnerships, we are on a journey of continuousinnovation. Ourenterprise-level experience in engineering enables us to offer critical development services fororganizations to besuccessful in the digital ages. Moreover, at Celestial, we focus on adhering to the complete softwaredevelopmentlifecycle process. Our experts are curious to work with the latest technologies and are extremelypassionate aboutsoftware development. Whether its Frontend Development, Backend, DevOps, QA and Testing or CloudHosting, ourtailor-made technology solutions, provide answers to the problems.We are ISO certified enterprise-grade software company.

Roles and Responsibilities

Celestial is rapidly expanding or Enterprise Data Services team to offer our customers a single point for all requirements including, Data Engineering, Data Analytics, Data Protection and Data Ethics/Governance.

We are looking for Senior Data Pipeline Engineers (multiple positions) for ongoing projects, within the Data Services and ML/AI Group. If you are motivated and willing to work in a startup mode on exciting NextGen projects do read on.

Requirements

• 4+ years of experience in big data engineering, with a focus on Spark and SQL.

• 4+ Years Experience in developing Large-scale ETL Pipelines using AWS services not limited to IAM, Glue, Lambda, S3, Athena, SNS, SQS, DynamoDB, RDS, EMR, ECS, Route 53 and Redshift.

• Highly skilled in Python, PostgreSQL/RDS/Redshift, and writing complex SQL queries and analysis of data correlations.

• Design, construct, and manage Data Lake environments including Data ingestion, staging, Data quality monitoring, and business modeling, Data Schema Normalization, etc.

• Develop, construct, secure, test and maintain architectures (databases and large-scale processing systems)

• Design, develop and implement technical solutions in SQL environment (databases, SQL, data marts, integrations)

• Experience in tools like Jenkins, GIT and/or AWS CodePipeline/CodeBuild/CodeDeploy

• Experience building data-driven unit test suites for data platforms and modeling highly dimensional datasets

• Strong functional understanding of Sagemaker, Jupyter and other Data Science tools on AWS

PREFERRED SKILLS

Experience designing and building big data architectures using tools such as Hadoop, Hive, or Kafka.

ADDITIONAL NOTES

If you are applying for Part-time/Contract role please email your resume with contact details, and "Hourly Consulting Rate Expectations".

Apply for this Job

Drop your Resume *


Attach or Drop Your resume
#J-18808-Ljbffr

  • Vancouver, Canada Celestial Systems Full time

    Celestial Systems, a name synonymous with Enterprise Softwaredevelopment since 2001. By designingreliable software,delivering quality services, and developing valuable partnerships, we are on a journey of continuousinnovation. Ourenterprise-level experience in engineering enables us to offer critical development services fororganizations to besuccessful in...


  • Vancouver, Canada Celestial Systems Full time

    Celestial Systems, a name synonymous with Enterprise Softwaredevelopment since 2001. By designingreliable software,delivering quality services, and developing valuable partnerships, we are on a journey of continuousinnovation. Ourenterprise-level experience in engineering enables us to offer critical development services fororganizations to besuccessful in...


  • Vancouver, Canada Celestial Systems Full time

    Celestial Systems, a name synonymous with Enterprise Softwaredevelopment since 2001. By designingreliable software,delivering quality services, and developing valuable partnerships, we are on a journey of continuousinnovation. Ourenterprise-level experience in engineering enables us to offer critical development services fororganizations to besuccessful in...


  • Vancouver, BC, Canada Celestial Systems Full time

    Celestial Systems, a name synonymous with Enterprise Softwaredevelopment since 2001. By designingreliable software,delivering quality services, and developing valuable partnerships, we are on a journey of continuousinnovation. Ourenterprise-level experience in engineering enables us to offer critical development services fororganizations to besuccessful in...

  • Senior Data Engineer

    2 weeks ago


    Vancouver, BC, Canada Zortechsolutions Full time

    Role: Data Engineer-SQL, Python, ETL Duration: Full Time We are looking for Senior Data Engineering Candidates based out of the Canada. Expertise in SQL, Python and ETL flows is a must have. Medium expertise/working knowledge in Scala is also needed, as our data pipelines are written in both Scala and Hive/Spark SQL, with new pipelines being written in...


  • Vancouver, Canada Fraser Health Full time

    Salary range The salary range for this position is CAD $45.46 - $65.35 / hour Why Fraser Health? Fraser Health is responsible for the delivery of hospital and community-based health services to over 1.9 million people in 20 diverse communities from Burnaby to Fraser Canyon on the traditional territories of the Coast Salish and Nlaka'pamux Nations....


  • Vancouver, Canada Amplitude Full time

    Amplitude is a leading digital analytics platform that helps companies unlock the power of their products. More than 2,300 customers, including Atlassian, Jersey Mike’s, NBCUniversal, Shopify, and Under Armour, rely on Amplitude to gain self-service visibility into the entire customer journey. Amplitude guides companies every step of the way as they...


  • Vancouver, BC, Canada railtown.ai Full time

    As a Senior Machine Learning Engineer, you will work with our growing AI team in developing AI and machine learning (ML) solutions for problems in the software development domain. This position will provide a mix of research and engineering problems as you will be involved in the development of the end-to-end machine learning pipeline, ranging from data...

  • Senior Data Engineer

    2 weeks ago


    Vancouver, BC, Canada Knewin Full time

    We are at the forefront of geolocation, cybersecurity, and anti-fraud innovation, developing and delivering cutting-edge technologies to help ensure regulatory compliance, combat bad online actors, alleviate user friction, and protect businesses from fraud. Achieving significant business and revenue growth over the past three years and dubbed a tech...


  • Vancouver, Canada Zortechsolutions Full time

    Role: Data Engineer-SQL, Python, ETLLocation: Vancouver/CANADADuration: Full TimeJob DescriptionKey Skills required: We are looking for Senior Data Engineering Candidates based out of the Canada. Expertise in SQL, Python and ETL flows is a must have. Medium expertise/working knowledge in Scala is also needed, as our data pipelines are written in both Scala...


  • Vancouver, Canada Zortechsolutions Full time

    Role: Data Engineer-SQL, Python, ETLLocation: Vancouver/CANADADuration: Full TimeJob DescriptionKey Skills required: We are looking for Senior Data Engineering Candidates based out of the Canada. Expertise in SQL, Python and ETL flows is a must have. Medium expertise/working knowledge in Scala is also needed, as our data pipelines are written in both Scala...


  • Vancouver, Canada Zortechsolutions Full time

    Role: Data Engineer-SQL, Python, ETLLocation: Vancouver/CANADADuration: Full TimeJob DescriptionKey Skills required: We are looking for Senior Data Engineering Candidates based out of the Canada. Expertise in SQL, Python and ETL flows is a must have. Medium expertise/working knowledge in Scala is also needed, as our data pipelines are written in both Scala...

  • Data Engineer

    3 weeks ago


    Vancouver, BC, Canada Fraser Health Full time

    Salary range The salary range for this position is CAD $37.01 - $53.20 / hour Why Fraser Health? Fraser Health is responsible for the delivery of hospital and community-based health services from Burnaby to Fraser Canyon on the traditional, ancestral and unceded territories of the Coast Salish and Nlaka’pamux Nations and is home to six Métis Chartered...


  • Vancouver, Canada Vancouver Airport Authority Full time

    YVR is made up of a team of diverse people who are working collaboratively to Connect BC Proudly to the World. Safety is at the core of everything we do; we’re innovative, fun, and we invest in our people. We’re a BC Top Employer for 14 years standing, with high engagement scores, an abundance of learning and development opportunities, and a holistic...


  • Vancouver, Canada Vancouver Airport Authority Full time

    YVR is made up of a team of diverse people who are working collaboratively to Connect BC Proudly to the World. Safety is at the core of everything we do; we’re innovative, fun, and we invest in our people. We’re a BC Top Employer for 14 years standing, with high engagement scores, an abundance of learning and development opportunities, and a holistic...


  • Vancouver, Canada Vancouver Airport Authority Full time

    YVR is made up of a team of diverse people who are working collaboratively to Connect BC Proudly to the World. Safety is at the core of everything we do; we’re innovative, fun, and we invest in our people. We’re a BC Top Employer for 14 years standing, with high engagement scores, an abundance of learning and development opportunities, and a holistic...

  • Lead Engineer

    7 days ago


    Vancouver, Canada Zonda Full time

    Job Title: Lead Engineer (Data) Location: Vancouver, BC / GVRD Reports To: Director of Data Engineering Job Summary: We are seeking a highly skilled and motivated individual to join our team as a Lead Engineer in Data Operation. As a Lead Engineer in Data Operations, you will be responsible for driving the development of modern data pipelines and...

  • Lead Engineer

    7 days ago


    Vancouver, Canada Zonda Full time

    Job Title: Lead Engineer (Data) Location: Vancouver, BC / GVRD Reports To: Director of Data Engineering Job Summary: We are seeking a highly skilled and motivated individual to join our team as a Lead Engineer in Data Operation. As a Lead Engineer in Data Operations, you will be responsible for driving the development of modern data pipelines and...

  • Lead Engineer

    4 days ago


    Vancouver, Canada Zonda Full time

    Job Title: Lead Engineer (Data) Location: Vancouver, BC / GVRD Reports To: Director of Data Engineering Job Summary: We are seeking a highly skilled and motivated individual to join our team as a Lead Engineer in Data Operation. As a Lead Engineer in Data Operations, you will be responsible for driving the development of modern data pipelines and...


  • Vancouver, Canada Industrial Light & Magic Full time

    Job Summary: This role is considered Hybrid, which means the employee will work 2-3 days onsite at a Company designated location and occasionally from home. Position Summary A Pipeline TD at Industrial Light & Magic provides direct support to productions and artists. They provide general troubleshooting of pipeline tools in a fast-paced, feature film visual...