Hadoop Developer

2 weeks ago

Toronto, Ontario, Canada Tekgence Inc Full time $120,000 - $150,000 per year

Hello,

Please find the Job Description below

Aws Hadoop Developer

Toronto ON

Role and Responsibilities

  • Understand requirement from product owners and translate into requirement and scope documents Decide on the best fitment of technologies services that are available in scope
  • Create Solution for data ingestion and data transformation using Hadoop ser-vices like Spark, Spark streaming, Hive, etc.
  • Create technical design documents to communicate solutions to the team and mentor the team to develop the solution Build the solution with Hadoop Services as per design specifications Assist team teams to build test cases and support with testing the solution Coordinate with Upstream, Downstream and other supporting teams for production implementation Provide post-production support for Solutions implemented Develop data engineering frameworks in Spark on AWS Data Lake platform Coordinate with clients, data users and key stakeholders to understand feature requirements needed merge them to create reusable design patterns
  • Data onboarding using the developed frameworks Understand and make sense of available code in Netezza and Hadoop to design a best way to implement its current features in AWS Data Lake Unit test code and aid with QASITP erf testing Migration to production environment

MUST HAVE skills and experience for this requirement. Please include skills related to technical as well as domain and non-technical skills and experience as applicable to the position.

  • Candidate should have strong working experience with Hadoop platform.
  • Strong hands-on experience on Hive, Spark with Scala.
  • In-depth knowledge and extensive experience in building batch workloads on Hadoop.
  • Adept in analyzing and refining requirements, consumption query patterns and choosing the right technology fit like RDBMS, data lake and data warehouse.
  • Should have knowledge of analytical data modelling on any of the RDBMS platform Hive Should have working experience in Pentaho Proven practical experience in migrating RDBMS based data to Hadoop on-prem 7 plus years of data experience in Data warehouse and Data Lake platforms
  • At least 2 years of implementation experience in AWS Data Lake, S3, EMR Glue, Python, AWS RDS, Amazon Redshift, Amazon Lake Formation, Airflow, Data Models, etc.
  • MUST have very strong knowledge of Pyspark Must understand data Encryption techniques and be able to implement them.
  • Must have experience in working with Bitbucket, Artifactory, AWS Code Pipeline Please list 5 NICE TO HAVE but not mandatory skills and experience for this requirement
  • Hands-on experience working with Terra bytes peat bytes scale data and millions of transactions per day.

Skills

  • to develop ETL pipe-line using Airflow Knowledge of Spark streaming or any other streaming jobs Ability to deploy code using AWS Code Pipeline and Bit bucket is an added plus
  • Expert in any of the following program-ming language Scala, Java and comfortable with working on Linux platform. Knowledge of Pythonic pipeline design AWS Cloud Infrastructure for services like S3, Glue, Secrets manager, KMS, Lambda, etc

  • Hadoop Architect

    5 days ago

    Toronto, Ontario, Canada Continental Technology Solutions Full time $120,000 - $180,000 per year

    Company Description Continental Technology Solutions is Our consulting and managed services deliver the resources and capabilities required to assess,design, build and manage an agile, flexible IT environment. You'll work with one dedicated team from strategy and assessment through design, implementation, management and optimization to reach a true...

  • Big Data Developer

    7 days ago

    Toronto, Ontario, Canada ehub global Full time $72,000 - $96,000 per year

    Role-Big-Data developerLocation- Toronto, ON (Onsite from day 1)JD:Experience: 8+ yearsBig Data: Apache Spark, Hadoop ecosystem(HDFS, Hive, Yarn).Programming : Expert in Scala and JavaBackend :Spring Boot, RESTful APIsDatabase : Microsoft SQL Server(MSSQL)Nice to have : Experience in migrating on-premise applications to cloud (Azure/AWS), familiarity with...

  • ETL Developer

    7 days ago

    Toronto, Ontario, Canada fc71e8c5-deda-48e9-b38c-23c0611c543b Full time $70,000 - $120,000 per year

    ResponsibilitiesDesign, develop, and optimize scalable ETL workflows to extract data from various sources such as relational databases, cloud storage (AWS, Azure Data Lake), and APIs (RESTful API).Transform raw data into structured formats suitable for analysis and reporting, ensuring accuracy and consistency.Maintain and enhance existing ETL pipelines using...

  • Sr. Java Developer

    2 weeks ago

    Toronto, Ontario, Canada TD Full time $76,800 - $115,200

    Work Location:Toronto, Ontario, CanadaHours:37.5Line of Business:Technology SolutionsPay Details:$76,800 - $115,200 CADTD is committed to providing fair and equitable compensation opportunities to all colleagues. Growth opportunities and skill development are defining features of the colleague experience at TD. Our compensation policies and practices have...

  • Big Data Developer

    2 weeks ago

    Toronto, Ontario, Canada Info Way Solutions Full time $80,000 - $120,000 per year

    Big Data Developer: (No of position – 1)JD:• Experience: 8+ years• Big Data: Apache Spark, Hadoop ecosystem(HDFS, Hive, Yarn).• Programming : Expert in Scala and Java• Backend :Spring Boot, RESTful APIs• Database : Microsoft SQL Server(MSSQL)• Nice to have : Experience in migrating on-premise applications to cloud (Azure/AWS), familiarity with...

  • Sr Full Stack Developer, GFT

    1 week ago

    Toronto, Ontario, Canada RBC Full time $80,000 - $120,000 per year

    Job DescriptionWhat Is The Opportunity?We are looking for a top notch software engineer who is a great team player and obsessive about the quality of their code and delivered service. This role requires someone who can roll up their sleeves and get the job done. The role requires a highly skilled individual with a strong background in software development,...

  • Big Data Developer

    7 days ago

    Toronto, Ontario, Canada fc71e8c5-deda-48e9-b38c-23c0611c543b Full time $70,000 - $120,000 per year

    Job OverviewWe are seeking a dynamic and innovative Big Data Developer to join our data engineering team. In this role, you will harness the power of large-scale data processing and analytics to drive strategic insights and support business growth. Your expertise in big data technologies, cloud platforms, and data integration will be crucial in designing,...

  • Lead Full Stack Developer, GFT

    1 week ago

    Toronto, Ontario, Canada RBC Full time $120,000 - $180,000 per year

    Job DescriptionWhat Is The Opportunity?We are looking for a top notch software engineering lead who are great team players and obsessive about the quality of their code and delivered service. This role requires someone who can roll up their sleeves and get the job done. The role requires a highly skilled individual with a strong background in software...

  • Senior Software Developer

    7 days ago

    Toronto, Ontario, Canada Electric Mind Full time $80,000 - $120,000 per year

    Electric Mind, formerly Intelliware, is a business and technology consulting leader in providing innovative, practical solutions driven by engineering excellence. Join our team to work on everything from front-end to server-side code and database architecture. We have an exceptional 30+ year delivery track record, deep technical expertise and disciplined...

  • Senior Developer

    6 hours ago

    Toronto, Ontario, Canada Market IQ Full time $100,000 - $150,000 per year

    Senior Back End DeveloperWe are disrupting the world of Finance – think Big Data; think Natural Language Processing; think Artificial Intelligence; think Slick User Experience.ResponsibilitiesOwn a variety of mission-critical and data-intensive problems, solve the tough problemsTranslate high-level business needs into well-defined and scoped...