Etl Specialist

1 week ago

Toronto, Ontario, Canada Maarut Inc Full time
  • Responsibilities Works in partnership with clients, advising them on information technology in order to meet their business objectives or overcome problems, work to improve structure and efficiency of an organization's I&IT systems. The I&IT Consultant may be used to provide strategic guidance to organizations with regard to Information Management & IT technology, IT infrastructures and the enablement of major business processes through enhancements to IT. Provides subject matter expertise in their field and highly expert technical assistance. General Skills Acts as the technical advisor/expert on all aspects of a specific deliverable Provide the quality assurance/quality control of specific deliverables Anticipates and resolves problems to ensure that the deliverables are completed within budget, to the highest quality, meeting or exceeding expectations Develops processes and procedures for implementing deliverables Prepares reports and presentations including options, recommendations, implementation plans, etc. Works with clients to define the scope of a project and to determine requirements Defines software, hardware and network requirements Analyzes I&IT requirements giving independent and objective advice on the use of I&IT Designs, tests, installs and monitors new systems and develops solutions and implementation of new systems Familiar with change-management principles and methodology Knowledge and understanding of Information Management principles, concepts, policies and practices

Additional Responsibilities

This role will focus on data architecture, data warehousing, data lakes, and analytics. The individual will be designing, developing, maintaining, and optimizing ETL (Extract, Transform, Load) processes in Databricks for data warehousing, data lakes, and analytics. The individual will work closely with data architects and business teams to ensure the efficient transformation and movement of data to meet business needs, including handling Change Data Capture (CDC) and streaming data.

  • Review business requirements, familiarize with and understand business rules and transactional data model
  • Define conceptual, logical model and physical model mapping from data source to curated model and data mart.
  • Analyze requirements and recommend changes to the physical model.
  • Develop scripts for the physical model, create database and/or delta lake file structure.
  • Access Oracle DB environments, set necessary tools for developing solution.
  • Implement data design methodologies, historical and dimensional models
  • Perform data profiling, assess data accuracy, design and document data quality and master data management rules
  • Functionality Review, Data Load review, Performance Review, Data Consistency checks.
  • Help troubleshooting data mart design issues
  • Review performance of ETL with developers and suggest improvements
  • Participate in end-to-end integrated testing for Full Load and Incremental Load and advise on issues

Tools used are:

  • Azure Databricks, Delta Lake, Delta Live Tables, and Spark to process structured and unstructured data.
  • Azure Databricks/PySpark (good Python/PySpark knowledge required) to build transformations of raw data into curated zone in the data lake.
  • Azure Databricks/PySpark/SQL (good SQL knowledge required) to develop and/or troubleshoot transformations of curated data into FHIR.

Requirements

Data design

  • Understand the requirements. Recommend changes to models to support ETL design.
  • Define primary keys, indexing strategies, and relationships that enhance data integrity and performance across layers.
  • Define the initial schemas for each data layer
  • Assist with data modelling and updates of source-to-target mapping documentation
  • Document and implement schema validation rules to ensure incoming data conforms to expected formats and standards
  • Design data quality checks within the pipeline to catch inconsistencies, missing values, or errors early in the process.
  • Proactively communicate with business and IT experts on any changes required to conceptual, logical and physical models, communicate and review timelines, dependencies, and risks.

Development of ETL strategy and solution for different sets of data modules

  • Understand the Tables and Relationships in the data model.
  • Create low level design documents and test cases for ETL development.
  • Implement error-catching, logging, retry mechanisms, and handling data anomalies.
  • Create the workflows and pipeline design
  • Development and testing of data pipelines with Incremental and Full Load.

Develop high quality ETL mappings/scripts/notebooks

  • Develop and maintain pipeline from Oracle data source to Azure Delta Lakes and FHIR
  • Perform unit testing
  • Ensure performance monitoring and improvement

Performance review, data consistency checks

  • Troubleshoot performance issues, ETL issues, log activity for each pipeline and transformation.
  • Review and optimize overall ETL performance.

End-to-end integrated testing for Full Load and Incremental Load

  • Plan for Go Live, Production Deployment.
  • Create production deployment steps.
  • Configure parameters, scripts for go live. Test and review the instructions.
  • Create release documents and help build and deploy code across servers.

Go Live Support and Review after Go Live.

  • Review existing ETL process, tools and provide recommendation on improving performance and reduce ETL timelines.
  • Review infrastructure and remediate issues for overall process improvement

Knowledge Transfer to Ministry staff, development of documentation on the work completed.

  • Document work and share the ETL end-to-end design, troubleshooting steps, configuration and scripts review.
  • Transfer documents, scripts and review of documents to Ministry.

Must Have Skills

  • 7+ years using ETL tools such as Microsoft SSIS, stored procedures, T-SQL
  • 2+ Delta Lake, Databricks and Azure Databricks pipelines
  • Strong knowledge of Delta Lake for data management and optimization.
  • Familiarity with Databricks Workflows for scheduling and orchestrating tasks.
  • 2+ years Python and PySpark
  • Solid understanding of the Medallion Architecture (Bronze, Silver, Gold) and experience implementing it in production environments.
  • Hands-on experience with CDC tools (e.G., GoldenGate) for managing real-time data.
  • SQL Server, Oracle

  • ETL Specialist

    7 days ago

    Toronto, Ontario, Canada Rubicon Path Full time

    Key ResponsibilitiesThe successful candidate will be responsible for:Designing, developing, and implementing ETL solutions using Azure Data FactoryWorking closely with cross-functional teams to ensure optimal solution integrationCollaborating with IT specialists and unit leads and working closely with support team members and relevant...

  • Python ETL Specialist

    12 hours ago

    Toronto, Ontario, Canada Professional Staffing Full time

    Job DescriptionWe are looking for a talented Python ETL Specialist to join our team. As a key member of our technical department, you will be responsible for designing, developing, and maintaining ETL systems that drive business value.Responsibilities:Design and implement ETL processes using Python.Work closely with cross-functional teams to ensure seamless...

  • Scalable ETL Pipeline Specialist

    7 days ago

    Toronto, Ontario, Canada Unity Full time

    We're seeking a skilled Scalable ETL Pipeline Specialist to join our team. As a Senior Machine Learning/MLOps Engineer, you'll focus on designing and building efficient ETL pipelines that can handle high volumes of data.Main ResponsibilitiesDesign and develop scalable ETL pipelines for real-time processing and batch processing.Collaborate with data...

  • Senior ETL Specialist

    6 days ago

    Toronto, Ontario, Canada Infotek Consulting Services Inc. Full time

    We are looking for a Senior ETL Specialist to join our team at Infotek Consulting Services Inc. The successful candidate will have 8+ years of experience in ETL development, with a focus on fraud initiatives.MUST HAVE:3+ years' experience on Azure services including Databricks, ADLS/Azure, SQL, PySpark notebooks ETL development3+ years hands-on experience is...

  • ETL Specialist

    1 week ago

    Toronto, Ontario, Canada Maarut Inc Full time

    Direct message the job poster from Maarut IncResponsibilitiesWorks in partnership with clients, advising them on information technology in order to meet their business objectives or overcome problems, work to improve structure and efficiency of an organization's I&IT systems. The I&IT Consultant may be used to provide strategic guidance to organizations with...

  • Enterprise ETL Developer

    3 days ago

    Toronto, Ontario, Canada Wirehead Full time

    Salary and Benefits:The Senior Data Transformation Specialist will be compensated competitively based on experience and qualifications. In addition to a competitive salary, the successful candidate will receive a comprehensive benefits package, including health and dental insurance, retirement savings plan, and paid time off.Key ResponsibilitiesDesign,...

  • ETL Tester

    2 weeks ago

    Toronto, Ontario, Canada Iris Software Inc. Full time

    Direct message the job poster from Iris Software Inc.Sr Associate - Hiring Specialist at Iris Software Inc.Iris's client, one of the Top 5 Banks in Canada, is looking to hire a Sr. ETL Tester for a long-term contract opportunity.Our Client is a Canadian multinational financial services company and the largest bank in Canada by market capitalization. The bank...

  • Senior Business Systems Analyst

    19 hours ago

    Toronto, Ontario, Canada S.i. Systems Full time

    About S.i. Systems: We are a leading provider of data integration solutions, and we are seeking a highly skilled Senior Business Systems Analyst - ETL Specialist to join our team.Job Description:Lead and manage the conversion of data from legacy systems to new platforms using industry-leading tools and technologies.Design, implement, and validate ETL...

  • ETL Process Specialist

    8 hours ago

    Toronto, Ontario, Canada Tata Consultancy Services Full time

    About Our Organization:Tata Consultancy Services is an IT services, consulting, and business solutions organization that has been partnering with many of the world's largest businesses in their transformation journeys for over 55 years. We have a unique Location Independent Agile delivery model, recognized as a benchmark of excellence in software...

  • ETL Developer

    13 hours ago

    Toronto, Ontario, Canada Tata Consultancy Services Full time

    About UsTata Consultancy Services (TCS) is a global leader in IT services, consulting, and business solutions. We partner with businesses to drive their digital transformation and help them achieve their goals.We believe in creating a workplace that is inclusive and diverse, where everyone feels valued and respected. Our commitment to culture and diversity...

  • RQ08927 Business Intelligence Specialist

    1 week ago

    Toronto, Ontario, Canada Randstad Canada Full time

    Are you an experienced ETL Developer with expertise in Azure Data Lake, Databricks, Python, and ETL tools like SSIS? Do you have hands-on experience building data ingestion pipelines, designing data warehouses, and optimizing data transformations? Our client seeks a Business Intelligence Specialist to design, develop, and implement scalable ETL solutions...

  • Senior ETL Specialist

    7 days ago

    Toronto, Ontario, Canada Messagepoint Inc. Full time

    The RoleWe are looking for a skilled Senior Software Developer to join our dynamic team as an AWS Glue expert. In this role, you will introduce AWS Glue as the primary ETL tool in our technology stack, configuring crawlers, data catalogs, and ETL jobs to automate schema discovery, normalization, and metadata management.You will establish best practices and...

  • ETL Automation Tester

    2 weeks ago

    Toronto, Ontario, Canada Iris Software Inc. Full time

    Iris's direct client, one of the Top 5 Bank in Canada, is looking to hire a ETL Automation Tester Role for a long term opportunity at Toronto, ON (Hybrid Position). Please respond back with your most recent resume if you would be interested..Our Client is a Canadian multinational financial services company and the largest bank in Canada by market...

  • ETL Data Integration Specialist

    6 days ago

    Toronto, Ontario, Canada Wirehead Full time

    Job SummaryAs an experienced Data Transformation Expert, you will be responsible for designing, developing, and implementing data transformation processes to extract insights from large datasets. This role requires a strong understanding of ETL development principles, database technologies, and business process knowledge.Key Responsibilities:Design, develop,...

  • RQ08964 Business Intelligence Specialist

    4 days ago

    Toronto, Ontario, Canada Randstad Canada Full time

    The Business Intelligence Specialist (ETL Developer) is responsible for designing, developing, and implementing robust data ingestion pipelines from Oracle sources to Azure Data Lake and Databricks environments. This role involves hands-on development using tools like Azure Data Factory, SSIS, SQL Server, and Python/PySpark to transform raw data into curated...

  • Informatica ETL Specialist

    7 days ago

    Toronto, Ontario, Canada SA TECHNOLOGIES Full time

    SA Technologies is a leading IT consulting firm with operations in the US, Canada, Mexico & India. As an Oracle Gold Partner, SAP Services Partner & IBM Certified enterprise, we provide top-notch services to our clients.We are seeking a highly skilled Informatica ETL Developer to join our team. The ideal candidate will have a strong background in data...

  • Etl Automation Tester

    2 weeks ago

    Toronto, Ontario, Canada Iris Software Inc. Full time

    Iris's direct client, one of the Top 5 Bank in Canada, is looking to hire a ETL Automation Tester Role for a long term opportunity at Toronto, ON (Hybrid Position) . Please respond back with your most recent resume if you would be interested.. Our Client is a Canadian multinational financial services company and the largest bank in Canada by market...

  • ETL Automation Tester

    3 days ago

    Toronto, Ontario, Canada Iris Software Inc. Full time

    Iris's direct client, one of the Top 5 Bank in Canada, is looking to hire a ETL Automation Tester Role for a long term opportunity at Toronto, ON (Hybrid Position) . Please respond back with your most recent resume if you would be interested.. Our Client is a Canadian multinational financial services company and the largest bank in Canada by market...

  • Informatica ETL Solutions Specialist

    16 hours ago

    Toronto, Ontario, Canada TechDigital Group Full time

    Job DescriptionAt TechDigital Group, we are seeking a skilled IICS Developer to lead our data integration efforts. This role is responsible for implementing strong ETL solutions, utilizing Informatica 9.x and above. The ideal candidate will have experience in application integration using Informatica, creating mappings with various transformations, and...

  • ETL Developer

    23 hours ago

    Toronto, Ontario, Canada S.i. Systems Full time

    Job DescriptionWe are looking for a skilled Data Pipeline Specialist to design, implement, and maintain complex data pipelines using SQL.The ideal candidate will have experience with data warehousing, ETL processes, and database design.This role is perfect for someone who enjoys working with data and wants to make a real impact in the industry.