Senior Big Data Hadoop ML Engineer

4 weeks ago

Old Toronto, Canada Rackspace Full time

About the Role:

We are seeking a highly skilled and experienced Senior Big Data Engineer to join our team. The ideal candidate will have a strong background in developing batch processing systems, with extensive experience in the Apache Hadoop ecosystem (Map Reduce, Oozie, Hive, Pig, HBase, Storm). This role involves working in Java and on Machine Learning pipelines for data collection or batch inference. This is a remote position, requiring excellent communication skills and the ability to solve complex problems independently and creatively.

Work Location: US-Remote

Key Responsibilities:

  • Develop scalable and robust code for large scale batch processing systems using Hadoop, Oozie, Pig, Hive, Map Reduce, Spark (Java), Python, Hbase
  • Develop, manage, and maintain batch pipelines supporting Machine Learning workloads
  • Leverage GCP for scalable big data processing and storage solutions
  • Implement automation/DevOps best practices for CI/CD, IaC, etc.

Requirements:

  • Proficiency in the Hadoop ecosystem with Map Reduce, Oozie, Hive, Pig, HBase, Storm
  • Strong programming skills with Java, Python, and Spark
  • Knowledge in public cloud services, particularly in GCP.
  • Experienced in Infrastructure and Applied DevOps principles in daily work. Utilize tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform to automate and improve development and release processes.
  • Ability to tackle complex challenges and devise effective solutions. Use critical thinking to approach problems from various angles and propose innovative solutions.
  • Worked effectively in a remote setting, maintaining strong written and verbal communication skills. Collaborate with team members and stakeholders, ensuring clear understanding of technical requirements and project goals.
  • Proven experience in engineering batch processing systems at scale.
  • Hands-on experience in public cloud platforms, particularly GCP. Additional experience with other cloud technologies is advantageous.

Must Have:

  • Experience with batch pipelines supporting Machine Learning workloads
  • Strong experience in programming language such as Java
  • Strong experience in the Apache Hadoop ecosystem
  • 10+ years of experience in customer-facing software/technology or consulting
  • 5+ years of experience with "on-premises to cloud" migrations or IT transformations
  • Technical degree: Computer Science, Software Engineering or related

Good to Have:

  • Familiarity with Terraform
  • Familiarity with Python
  • 5+ years of experience building and operating solutions built on GCP

  • Senior Big Data Hadoop ML Engineer

    3 months ago

    Old Toronto, Canada Rackspace Full time

    About the Role:We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing batch processing systems, with extensive experience in the Apache Hadoop ecosystem (Map Reduce, Oozie, Hive, Pig, HBase, Storm). This role involves working in Java and on Machine...

  • Senior Big Data Hadoop ML Engineer

    3 months ago

    Toronto, Canada Apollo Full time

    About the Role:We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing batch processing systems, with extensive experience in the Apache Hadoop ecosystem (Map Reduce, Oozie, Hive, Pig, HBase, S

  • Senior Big Data Engineer

    4 weeks ago

    Toronto, Ontario, Canada Astra North Infoteck Inc. Full time

    Astra North Infoteck Inc. is seeking a highly skilled Solution Architect to lead the design and implementation of our cloud-based data solutions. The ideal candidate will have extensive experience with Java, Spark, Hadoop, and Big Data technologies, as well as a strong understanding of case management systems.Key Responsibilities:Design and implement...

  • Big Data Lake Hadoop Cloudera Expert

    6 months ago

    Toronto, Canada CB Canada Full time

    Big Data Lake Hadoop Cloudera Expert On behalf of our client in the Banking Sector, PROCOM is looking for a Big Data Lake Hadoop Cloudera Expert. Big Data Lake Hadoop Cloudera Expert – Job Description Directs development of complex artifacts for digital media (web or native app) to exchange information, elicit emotion and facilitate interaction Data Lake...

  • Senior Big Data Developer Lead

    2 weeks ago

    Toronto, Ontario, Canada Astra North Infoteck Inc. Full time

    Job Title: Senior Big Data Developer LeadAstra North Infoteck Inc. is seeking a highly skilled Senior Big Data Developer Lead to join our team in designing and implementing cutting-edge big data solutions using Java, Spark, Hadoop, and Vue.js/node.js.About the Role:We are looking for a seasoned professional with 10+ years of experience in leading big data...

  • Data Engineering Lead

    1 week ago

    Old Toronto, Canada Rackspace Full time

    About the Role:We are seeking an experienced Data Engineering Lead to join our team at Rackspace. The ideal candidate will have a strong background in developing scalable batch processing systems, with extensive experience in the Apache Hadoop ecosystem (Map Reduce, Oozie, Hive, Pig, HBase, Storm). This role involves working on Machine Learning pipelines for...

  • Senior Software Engineer, AI/ML Platform Specialist

    4 weeks ago

    Toronto, Ontario, Canada Autodesk Full time

    Position OverviewWe are seeking a highly skilled software engineer to develop our next-generation AI/ML platform used in the development of Autodesk's suite of products and services. The ideal candidate will have a strong background in software development and engineering, with a focus on AI/ML technologies.ResponsibilitiesInnovative System Design: Lead the...

  • Senior Software Development Engineer for AI/ML Platform

    1 week ago

    Toronto, Ontario, Canada Autodesk Full time

    Job OverviewWe are seeking an experienced software development engineer to design and implement a cutting-edge AI/ML platform used in the development of Autodesk's suite of products and services.About UsWelcome to Autodesk, where we help innovators turn their ideas into reality. Our culture is built on a strong foundation of collaboration, innovation, and...

  • Big Data Lake Hadoop Cloudera Specialist

    4 weeks ago

    Toronto, Ontario, Canada CB Canada Full time

    Job Title: Big Data Lake Hadoop Cloudera SpecialistCB Canada is seeking a highly skilled Big Data Lake Hadoop Cloudera Specialist to join our team. The ideal candidate will have a strong background in big data technologies and experience in working with Hadoop and Cloudera.Key Responsibilities:Directs the development of complex artifacts for digital media to...

  • Senior Data Engineer

    1 month ago

    Old Toronto, Ontario, Canada Data Engineer Jobs Full time

    About This RoleWe are seeking a highly skilled Senior Data Engineer to join our Analytics Engineering team. As a key member of this team, you will be responsible for designing and building scalable data models and ETL pipelines to support business decisions.Key Responsibilities:Collaborate with data scientists to design data models and answer questions.Work...

  • Senior Data Engineer, Big Data

    4 weeks ago

    Toronto, Ontario, Canada Autodesk Full time

    Job OverviewAutodesk is seeking a Senior Data Engineer, Big Data to join our Enterprise Data Engineering & Visualisation (EDEV) team. The EDEV Team is responsible for building data pipelines and solutions that enable our users to gain valuable business insights. These solutions serve as the foundation for all data engineers, data scientists, and data...

  • Principal Software Engineer, AI/ML Platform

    2 months ago

    Old Toronto, Canada Autodesk Full time

    Job Requisition ID # 24WD82577 Position Overview We are seeking a dynamic and enthusiastic principal software engineer to develop our next-generation AI/ML platform used in the development of Autodesk’s suite of products and services. Join our dynamic and rapidly expanding team to help build innovative capabilities that enable faster and more secure...

  • Senior Data Engineer: Driving Business Growth with Data Insights

    1 week ago

    Old Toronto, Canada Data Engineer Jobs Full time

    As a Senior Data Engineer at Mozilla, you will play a pivotal role in shaping the company's data strategy and driving business growth through informed decision-making.About the RoleWe are seeking an experienced data engineer to join our Analytics Engineering team. In this role, you will work closely with data scientists to design and implement scalable data...

  • Senior Data Engineer, Big Data

    4 weeks ago

    Toronto, Ontario, Canada Autodesk Full time

    Job OverviewAutodesk is seeking a Senior Software Developer, Big Data to join our Enterprise Data Engineering & Visualisation (EDEV) team. The EDEV Team is responsible for building data pipelines and solutions that enable our users to gain valuable business insights. These solutions serve as the foundation for all data engineers, data scientists, and data...

  • Senior Data Scientist and Engineer

    4 weeks ago

    Toronto, Ontario, Canada Apollo Full time

    About the Role:We are seeking a highly skilled Senior Big Data Engineer to join our team at Apollo. The ideal candidate will have a strong background in developing batch processing systems and extensive experience in the Apache Hadoop ecosystem.Key Responsibilities:Design and develop high-performance batch processing systemsImplement data processing...

  • Big Data Engineer with Java

    2 months ago

    Toronto, Ontario, Ontario, Canada VLink Inc Full time

    Big Data Engineer with JavaToronto, ON, JR1017573FulltimeJob Description:We are seeking a highly skilled and experienced Big Data Engineer to join our dynamic team. The ideal candidate will have a solid background in application development with specific expertise in Hadoop, Spark, and cloud technologies. This role involves developing and optimizing big data...

  • Senior Software Engineer, AI/ML Platform Specialist

    1 month ago

    Toronto, Ontario, Canada Autodesk Full time

    Position OverviewWe are seeking a highly skilled software engineer to develop and lead the development of our next-generation AI/ML platform. This platform will be used in the development of Autodesk's suite of products and services. The ideal candidate will have a strong background in software development and engineering, with a focus on AI/ML...

  • Data Engineer with Expertise in Cloud and Big Data

    1 week ago

    Toronto, Ontario, Canada Astra North Infoteck Inc. Full time

    Job Title: Data Engineer with Expertise in Cloud and Big DataAstra North Infoteck Inc. is seeking an experienced Data Engineer to lead the development of our big data solutions.We are offering a competitive salary of $140,000 - $180,000 per year, depending on experience and qualifications.About the Role:We are looking for a skilled engineer who can design,...

  • Senior Data Engineer

    1 week ago

    Toronto, Ontario, Canada Scotiabank Full time

    About This RoleScotiabank, a leading financial institution, seeks an exceptional Senior Data Engineer to join its Client Central technology team. As a key member of this team, you will be responsible for designing, developing, and implementing large-scale data processing solutions using big data technologies and Python.Job DescriptionThe ideal candidate will...

  • Lead Data Architect in Big Data Engineering

    2 weeks ago

    Toronto, Ontario, Canada Royal Bank of Canada Full time

    Job OverviewRoyal Bank of Canada (RBC) is a leading Canadian bank seeking an experienced Manager, Data Engineering to join its Canadian Banking Operations Data and Analytics Center of Excellence. This role offers the opportunity to lead the development and maintenance of automated data pipelines and infrastructure.About the RoleWe are looking for a skilled...