Data Engineer

Found in: Appcast CA A2 P - 2 weeks ago

Toronto, Canada HCLTech Full time

Position: Data Engineer (Data Bricks)

Location: Toronto, Ontario / Remote


Skills Requirements: JD


Data Bricks:

• Strong hands on in Pyspark and Apache Spark

• Strong hands on in Medallion architecture

• Experience in Native Spark Migration to Databricks.

• Experience in Building Data Governance Solutions like Unity Catalog, Azure Purview etc.

•Highly experienced in Usability Optimization (Auto Compaction, ZOrdering, Vaccuming), Cost Optimization and Performance Optimization.

• Build Very Strong Orchestration Layer in Databricks/ADF…. Workflows.

• Build CICD for Databricks in Azure Devops.

• Process near Real time Data thru Auto Loader, DLT Pipelines.

• Implement Security Layer in Delta Lake.

• Implement Massive Parallel Processing Layers in Spark SQL and PySpark.

• Implement Cost effective Infrastructure in Databricks.

• Experience In extracting logic and from on prem layers, SAP, ADLS into Pyspark/ADLS using ADF/Databricks.


Azure Synapse Analytics/Azure data Factory (ADF):

- Hands on Experience in Azure Synapse Analytics, Azure Data Factory and Data Bricks, Azure Storage, Azure Key Vault, SQL Pools CI/CD Pipeline Designing and other Azure services like functions, logic apps

- Linked services, Various Runtimes, Datasets, Pipelines, Activities

- Strong Hands on Experience in Various Activites like Control flow logic and conditions (For Each, if, switch, until), Lookup, Stored procedure, scripts, validations, Copy Data, Data flow, Azure functions, Notebooks, SQL Pool Stored procedures and etc

- Strong hands on exp in deployment of code through out landscape (Dev -> QA -> Prod), Git Hub, CI/CD pipelines and etc


SQL Server stored procedures:


strong hands on creating the SQL stored procedures

- Functions, Stored Procedures, how to call one SP into another, How to process record-by-record

- Dynamic SQL


Python:

Must have strong background about the Python libraries like PySpark, Pandas, NumPy, pymysql, Oracle, Pyspark libraries

- Must have strong hands on to get data through APIs

- Must be able to install libraries and help users to troubleshoot issues

- Must have knowledge to get the data through stored procedures via Python

- Should be able to debug the Python code


Sparks:

- Hands on experioence in Spark Pools, PySpark

- Should be able to merge data/delta loads through Notebooks

- Must have strong background about the Python libraries and PySpark

  • Data Analyst

    Found in: Talent CA 2A C2 - 2 weeks ago

    Toronto, Canada Data SEA Consulting Full time

    Data SEA Consulting – Data Analyst Company DescriptionData SEA Consulting focuses on helping mid-sized businesses harness the power of their data by leveraging the latest analytics tools and techniques. We work with Private Equity Funds and Independent Business Owners to find the unique data solutions that will drive the most value for their business....

  • Data Analyst

    Found in: Appcast CA C2 Glassdoor - 2 weeks ago

    Toronto, Canada Data SEA Consulting Full time

    Data SEA Consulting – Data Analyst Company DescriptionData SEA Consulting focuses on helping mid-sized businesses harness the power of their data by leveraging the latest analytics tools and techniques. We work with Private Equity Funds and Independent Business Owners to find the unique data solutions that will drive the most value for their business....

  • Data Analyst

    Found in: Appcast CA A2 P - 2 weeks ago

    Toronto, Canada Data SEA Consulting Full time

    Data SEA Consulting – Data Analyst Company DescriptionData SEA Consulting focuses on helping mid-sized businesses harness the power of their data by leveraging the latest analytics tools and techniques. We work with Private Equity Funds and Independent Business Owners to find the unique data solutions that will drive the most value for their business....

  • Data Analyst

    Found in: Whatjobs CA C2 - 3 weeks ago

    Toronto, Canada Data SEA Consulting Full time

    Data SEA Consulting – Data Analyst Company Description Data SEA Consulting focuses on helping mid-sized businesses harness the power of their data by leveraging the latest analytics tools and techniques. We work with Private Equity Funds and Independent Business Owners to find the unique data solutions that will drive the most value for their business....

  • iOS Engineer, Observability SDK

    Found in: beBee S CA - 3 weeks ago

    Toronto, Canada Data Theorem Full time

    Data Theorem is an exciting company focused on creating a more secure world for data. Rooted in a strong engineer first culture, every employee has an impact on product and direction. We are searching for exceptional talent pursuing an opportunity to grow and take ownership of the projects that resonate most with them.As an iOS engineer, you will be...

  • iOS Engineer, Observability SDK

    Found in: beBee jobs CA - 2 weeks ago

    Toronto, Ontario, Canada Data Theorem Full time

    Data Theorem is an exciting company focused on creating a more secure world for data. Rooted in a strong engineer first culture, every employee has an impact on product and direction. We are searching for exceptional talent pursuing an opportunity to grow and take ownership of the projects that resonate most with them.As an iOS engineer, you will be...

  • Data Engineer

    Found in: Appcast CA C2 Glassdoor - 2 weeks ago

    Toronto, Canada Bitstrapped Full time

    We are looking for a hands-on Senior Data Engineer in a full-time salaried position. The Data Engineer will work with our Data Services Practice and work alongside our talented cloud architects to play a key role in our delivery teams. The Data Engineer will be involved in the design, programming, and implementation of data platform components and serve as a...

  • Data Engineer

    Found in: Appcast CA A2 P - 2 weeks ago

    Toronto, Canada Bitstrapped Full time

    We are looking for a hands-on Senior Data Engineer in a full-time salaried position. The Data Engineer will work with our Data Services Practice and work alongside our talented cloud architects to play a key role in our delivery teams. The Data Engineer will be involved in the design, programming, and implementation of data platform components and serve as a...

  • Data Engineer

    Found in: Whatjobs CA C2 - 2 weeks ago

    Toronto, Canada Bitstrapped Full time

    We are looking for a hands-on Senior Data Engineer in a full-time salaried position. The Data Engineer will work with our Data Services Practice and work alongside our talented cloud architects to play a key role in our delivery teams. The Data Engineer will be involved in the design, programming, and implementation of data platform components and serve...

  • Data Engineer

    Found in: Talent CA 2A C2 - 2 weeks ago

    Toronto, Canada Organigram Inc Full time

    Description:We are seeking a skilled and experienced Data Engineer to join our team. The ideal candidate will have expertise in SQL Server, Azure Data Factory, and data ingestion architecture within the Azure Cloud environment. Additionally, familiarity with Dynamics 365 and experience in IoT (Internet of Things) would be highly...

  • Data Engineer

    Found in: Appcast CA C2 Glassdoor - 2 weeks ago

    Toronto, Canada Organigram Inc Full time

    Description:We are seeking a skilled and experienced Data Engineer to join our team. The ideal candidate will have expertise in SQL Server, Azure Data Factory, and data ingestion architecture within the Azure Cloud environment. Additionally, familiarity with Dynamics 365 and experience in IoT (Internet of Things) would be highly...

  • Data Engineer

    Found in: Appcast CA A2 P - 2 weeks ago

    Toronto, Canada Organigram Inc Full time

    Description:We are seeking a skilled and experienced Data Engineer to join our team. The ideal candidate will have expertise in SQL Server, Azure Data Factory, and data ingestion architecture within the Azure Cloud environment. Additionally, familiarity with Dynamics 365 and experience in IoT (Internet of Things) would be highly...

  • Data Engineer

    Found in: Whatjobs CA C2 - 2 weeks ago

    Toronto, Canada Organigram Inc Full time

    Description: We are seeking a skilled and experienced Data Engineer to join our team. The ideal candidate will have expertise in SQL Server, Azure Data Factory, and data ingestion architecture within the Azure Cloud environment. Additionally, familiarity with Dynamics 365 and experience in IoT (Internet of Things) would be highly beneficial....

  • Data Engineer

    Found in: Talent CA C2 - 2 weeks ago

    Toronto, Canada SearchLabs Full time

    Job DescriptionTitle: Data EngineerType: ContractLocation: TorontoEnvironment: HybridWe are seeking a highly skilled and motivated Data Engineer to join our team. The ideal candidate will have extensive experience in designing, building, and maintaining scalable data pipelines and platforms using cloud-based technologies, particularly within the Microsoft...

  • Data Engineer

    Found in: Talent CA C2 - 2 weeks ago

    Toronto, Canada SearchLabs Full time

    Title: Data EngineerType: ContractLocation: TorontoEnvironment: HybridWe are seeking a highly skilled and motivated Data Engineer to join our team. The ideal candidate will have extensive experience in designing, building, and maintaining scalable data pipelines and platforms using cloud-based technologies, particularly within the Microsoft Azure ecosystem....

  • Data Engineer

    Found in: Talent CA C2 - 2 weeks ago

    Toronto, Canada Tiger Analytics Full time

    Tiger Analytics is a fast-growing advanced analytics consulting firm. Our consultants bring deep expertise in Data Science, Machine Learning and AI. We are the trusted analytics partner for several Fortune 100 companies, enabling them to generate business value from data. Our business value and leadership has been recognized by various market research firms,...

  • Data Engineer

    1 week ago

    Toronto, Canada hireVouch Full time

    Our client is a Canadian-based international real estate investor, developer, and property manager. They are growing their team and looking for a passionate Data Engineer to join.The Data Engineer will be responsible for building, processing, reviewing, and managing data alongside support from a data scientist. Enhance your skills working with ETL pipelines,...

  • Data Engineer

    6 days ago

    Toronto, Canada hireVouch Full time

    Our client is a Canadian-based international real estate investor, developer, and property manager. They are growing their team and looking for a passionate Data Engineer to join.The Data Engineer will be responsible for building, processing, reviewing, and managing data alongside support from a data scientist. Enhance your skills working with ETL pipelines,...

  • Data Engineer

    Found in: Appcast CA C2 Glassdoor - 1 week ago

    Toronto, Canada hireVouch Full time

    Our client is a Canadian-based international real estate investor, developer, and property manager. They are growing their team and looking for a passionate Data Engineer to join.The Data Engineer will be responsible for building, processing, reviewing, and managing data alongside support from a data scientist. Enhance your skills working with ETL pipelines,...

  • Data Engineer

    Found in: Appcast CA A2 P - 1 week ago

    Toronto, Canada hireVouch Full time

    Our client is a Canadian-based international real estate investor, developer, and property manager. They are growing their team and looking for a passionate Data Engineer to join.The Data Engineer will be responsible for building, processing, reviewing, and managing data alongside support from a data scientist. Enhance your skills working with ETL pipelines,...