Senior Data Engineer to create, manage, and monitor data pipelines using Azure Data Factory and[...]

3 weeks ago

Ottawa, Canada S I Systems Full time
Senior Data Engineer to create, manage, and monitor data pipelines using Azure Data Factory and Azure Synapse for a Government client

Our valued public sector client requires two Senior Data Engineers to create, manage, and monitor data pipelines using Azure Data Factory and Azure Synapse for a Government client.

Tasks:

  • Azure Data Factory (ADF) and Azure Synapse Analytics for data ingestion, transformation, and processing experience
  • Data integration and the development of data pipelines
  • Data processing for cost-effectiveness in Azure's scalable and flexible cloud environment
  • Creation, management, and optimization of data processing workflows within the Azure data platform for the clients data warehouse modernization initiative
  • Python scripting for automation of storage-related tasks, configuration management, and system monitoring
  • Manage workload using Azure Synapse for a data lake project.
  • Utilize Azure Synapse to distribute the workload (performance management)
  • Develop and maintain Python scriptsfor efficient data manipulation , migration, and integration with storage systems.
  • Develop spark applications in python (PySpark) on distributed environment
  • Creating Spark clusters and configuring high concurrency clusters
  • Create and maintain documentation for compliance and knowledge transfer
  • Design for scalability and optimize code for peak performance.
  • Develop test cases for unit and integration tests
  • Git branching strategies using Azure DevOps Repo & pipeline
  • Identify ways to improve data reliability, efficiency and quality
  • Use large data sets to address business issues
  • Deliver updates to stakeholders based on analytics

Must-haves:

  • 4+ years of experience using Azure Data Factory
  • 4+ years of experience using Azure Synapse
  • 2+ years of experience creating Git branching strategies using ADO Repo and Pipeline
  • 2+ years of Python development experience

Nice-to-haves:

  • Experience working for the Federal Government
  • Experience working with mainframe systems/files
#J-18808-Ljbffr

  • Senior Data Engineer to create, manage, and monitor data pipelines using Azure Data Factory and[...]

    3 weeks ago

    Ottawa, Canada S I Systems Full time

    Senior Data Engineer to create, manage, and monitor data pipelines using Azure Data Factory and Azure Synapse for a Government clientOur valued public sector client requires two Senior Data Engineers to create, manage, and monitor data pipelines using Azure Data Factory and Azure Synapse for a Government client.Tasks:Azure Data Factory (ADF) and Azure...

  • Senior Data Engineer to create, manage, and monitor data pipelines using Azure Data Factory and

    6 days ago

    Ottawa, Ontario, Canada S I Systems Full time

    Senior Data Engineer to create, manage, and monitor data pipelines using Azure Data Factory and Azure Synapse for a Government clientOur valued public sector client requires two Senior Data Engineers to create, manage, and monitor data pipelines using Azure Data Factory and Azure Synapse for a Government client.Tasks:Azure Data Factory (ADF) and Azure...

  • Data Engineer

    2 months ago

    Ottawa, Canada CGI Full time

    Position Description: Security clearance: Reliability Clearance a MUST Location: Ottawa, ON Remote: Yes Are you a data engineering professional interested in working within a talented and collaborative team developing Cloud, Data, and AI solutions for leading global clients? Do you often find yourself in discussions about disparate and scattered...

  • Data Engineer

    3 weeks ago

    Ottawa, Canada CGI Full time

    Position Description: Security clearance: Reliability Clearance a MUST Location: Ottawa, ON Remote: Yes Are you a data engineering professional interested in working within a talented and collaborative team developing Cloud, Data, and AI solutions for leading global clients? Do you often find yourself in discussions about disparate and scattered...

  • Data Engineer

    3 weeks ago

    Ottawa, Canada CGI Full time

    Position Description: Security clearance: Reliability Clearance a MUST Location: Ottawa, ON Remote: Yes Are you a data engineering professional interested in working within a talented and collaborative team developing Cloud, Data, and AI solutions for leading global clients? Do you often find yourself in discussions about disparate and scattered...

  • Data Engineer

    5 days ago

    Ottawa, Ontario, Canada CGI Full time

    Position Description: Security clearance: Reliability Clearance a MUST Location: Ottawa, ON Remote: Yes Are you a data engineering professional interested in working within a talented and collaborative team developing Cloud, Data, and AI solutions for leading global clients? Do you often find yourself in discussions about disparate and scattered data...

  • Data Engineer

    6 days ago

    Ottawa, Ontario, Canada Tomlinson Group of Companies Full time

    As a Data Engineer at Tomlinson, you will play a crucial role in collaborating with the Data Architect and AI team to develop and implement data integration strategies, build data pipelines, perform data modeling, ensure data governance, and plan scalable solutions. You will work with various business units to handle diverse data objects from disparate...

  • Data Engineer

    1 month ago

    Ottawa, Canada Tomlinson Group of Companies Full time

    As a Data Engineer at Tomlinson, you will play a crucial role in collaborating with the Data Architect and AI team to develop and implement data integration strategies, build data pipelines, perform data modeling, ensure data governance, and plan scalable solutions. You will work with various business units to handle diverse data objects from disparate...

  • Senior Data Engineer

    5 days ago

    Ottawa, Ontario, Canada CB Canada Full time

    Senior Data EngineerOn behalf of our public sector client, we are seeking a Senior Data Engineer. Remote work at present. Onsite work in Ottawa may be required at a later date.Job Description:The Senior Data Engineer Service Provider will be responsible for leading the delivery of data on data and analytics platform. These services are at the forefront of...

  • Senior Data Engineer

    2 months ago

    Ottawa, Canada CB Canada Full time

    Senior Data EngineerOn behalf of our public sector client, we are seeking a Senior Data Engineer. Remote work at present. Onsite work in Ottawa may be required at a later date.Job Description:The Senior Data Engineer Service Provider will be responsible for leading the delivery of data on data and analytics platform. These services are at the forefront of...

  • Senior Data Engineer

    3 weeks ago

    Ottawa, Canada CB Canada Full time

    Senior Data EngineerOn behalf of our public sector client, we are seeking a Senior Data Engineer. Remote work at present. Onsite work in Ottawa may be required at a later date.Job Description:The Senior Data Engineer Service Provider will be responsible for leading the delivery of data on data and analytics platform. These services are at the forefront of...

  • Manager Azure Data Engineering

    3 weeks ago

    Ottawa, Canada Avanade Full time

    Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly...

  • Manager Azure Data Engineering

    2 months ago

    Ottawa, Canada Avanade Full time

    Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly...

  • Manager Azure Data Engineering

    1 month ago

    Ottawa, Canada Avanade Full time

    Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly...

  • Manager Azure Data Engineering

    1 month ago

    Ottawa, Canada Avanade Full time

    Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly...

  • Manager Azure Data Engineering

    3 weeks ago

    Ottawa, Canada Avanade Full time

    Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly...

  • Manager Azure Data Engineering

    3 weeks ago

    Ottawa, Canada Avanade Full time

    Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly...

  • Manager Azure Data Engineering

    6 days ago

    Ottawa, Ontario, Canada Avanade Full time

    Our talented Data & AI Practice is made up of globally recognized experts - and there's room for more analytical and ambitious data professionals. If you're passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let's talk. Take your skills to a new level and launch a career where you can truly do...

  • Manager Azure Data Engineering

    1 month ago

    Ottawa, Canada Avanade Full time

    Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly...

  • Manager Azure Data Engineering

    1 month ago

    Ottawa, ON, Canada Avanade Full time

    Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly...