Lead Azure Data Engineer/Unity Catalog

2 weeks ago

Toronto, Ontario, Canada BeachHead Full time $120,000 - $180,000 per year

Working with one of the top financial clients, this role calls for a Lead Azure Data Engineer/Unity Catalog who will design, develop, and implement applications using in-demand languages and technologies (e.g., WebSphere, Informatica, etc.) to support business requirements.

Responsibilities:

  • Data Source Transformation: Migrate the current data source from a tactical infrastructure, which involves direct pulls from virtual machines (VMs), to a strategic infrastructure utilizing Azure Data Factory (ADF) and Databricks.
  • ETL Process Implementation: Develop an Extract, Transform, Load (ETL) process to efficiently transfer data from various sources, including SFTP, web services, and databases, into a Delta Lake. This will enable seamless data consumption for business analytics.
  • Connectivity Assistance: Support the business in establishing a robust connection between Plotly Dash and Databricks, ensuring that data visualizations are powered by the latest data insights.
  • Databricks Setup and Management: Establish the Unity Catalog, configure clusters, define schemas, and set up permissions within Databricks to ensure secure and organized data management.

Desired Skill Set:

  • 7+ Years experience with ADF, Databricks, Unity Catalog and Azure Functions (C#)
  • In-depth development experience with Python/notebook on Databricks/Spark/pandas
  • Experience in loading datasets from various sources using ADF to Databricks with Medallion architecture
  • Experience in converting a Databricks workspace from Hive metastore to Unity catalog & setup up associated permissions
  • Creating delta table schema/tables with proper design/properties & security permissions
  • DevOps & deployment pipeline using Azure DevOps/GitHub Enterprise (especially on databricks on cluster/secret management and code)
  • In-depth experience on Azure Entra ID for SSO/RBAC setup
  • In-depth experience in working with/deploying to Azure PaaS resources such as Azure SQL PaaS, ADF/SHIR, ADLS, Databricks, and Synapse dedicated SQL databases

Nice To Have:

  • Experience with Delta Live Table.
  • Familiar with working in lockdown windows environment for a highly regulated industry
  • Financial Services Industry experience

BeachHead is an equal opportunity agency and employer. We advocate for you and welcome anyone regardless of race, color, religion, national origin, sex, physical or mental disability, or age.

Privacy Policy

  • Azure Data Engineer

    7 days ago

    Toronto, Ontario, Canada Centraprise Full time $90,000 - $120,000 per year

    Azure Data Engineer (Terraform, Databricks, Python)Toronto, Canada12+ Months ContractJob Description:Databrick: S/He needs to change the pipelines, codes . Candidate has to have hands on experience on Databrick and UC (Unity Catalog).Azure: We are Microsoft shop, everything we have is running on Azure. S/He should be comfortable on Azure framework (ADF,...

  • Databricks Data Engineer

    7 days ago

    Toronto, Ontario, Canada Rubicon Path Full time $120,000 - $180,000 per year

    Who we are looking forThis is a hands-on Data Engineer position. We are looking for candidate with good knowledge on Bigdata technology and strong development experience with Databricks. What you will be responsible forAs Data Engineer you willDesign & Develop custom high throughput and configurable frameworks/librariesArchitect and implement scalable data...

  • Databricks Data Engineer

    2 hours ago

    Toronto, Ontario, Canada CloudTech Innovations Full time $120,000 - $180,000 per year

    Job Title: Data Engineer – DatabricksLocation:Onsite – Toronto, CanadaEmployment Type:ContractAbout the RoleWe are seeking an experienced Data Engineerwith a strong background inDatabricks,Apache Spark, andmodern cloud data platforms. The ideal candidate has over5 years of experiencedesigning, developing, and maintaining scalable data pipelines and...

  • Tech Lead Azure Application Services Databricks

    2 hours ago

    Toronto, Ontario, Canada Collabera Full time $3,000 - $32,000 per year

    Position Details:Role:Tech Lead Azure Application Services DatabricksContract:10 months (with possible extension)Weekly Hours:37.5Location:Toronto, ON- hybridHybrid:Core on-site 5 days/month - every Tuesday, one Friday a monthCompensation:$65-$80/hourInterview Process:2 rounds- 1st round: Virtual w/ Director, 30 mins and 2nd round: In-person (mandatory),...

  • Lead Azure Data engineer

    2 weeks ago

    Toronto, Ontario, Canada Itech Enterprises Full time $120,000 - $180,000 per year

    Role DescriptionThis is a full-time on-site role for a Lead Azure Data Engineer located in Toronto, ON. The Lead Azure Data Engineer will be responsible for designing, implementing, and managing data solutions on the Azure platform. Daily tasks include data engineering, data modeling, extract transform load (ETL) processes, and data warehousing. The...

  • Canada - Azure Applications Tech Lead

    5 hours ago

    Toronto, Ontario, Canada Collabera Full time $65,000 - $70,000 per year

    Position:Technical Lead (Azure Applications)Industry:BankingType:10 months contractLocation:Toronto, ONWork mode:5x days/month (every Tuesday and 1 Friday on-site)Interviews:1st round - Virtual w/ Director, 30 mins| 2nd round - In-person (mandatory), panel w/ Director & team, 30-60 minsPay Rate: $65 - $70 CAD per hourDay to day:Lead Azure AD/EntraID...

  • Data Engineering Lead

    2 weeks ago

    Toronto, Ontario, Canada Dew Software Full time $120,000 - $180,000 per year

    Job Title: Data Engineering Lead Location: Toronto, Ontario, Canada (Hybrid) Experience Level: 8–10 Years Employment Type: Contract - W2 OnlyPosition OverviewWe are seeking a highly skilled and motivatedData Engineering Leadto design, build, and optimize scalable data pipelines and architectures for enterprise analytics and business intelligence platforms....

  • Data Engineering Manager

    1 hour ago

    Toronto, Ontario, Canada TEEMA Full time

    Hybrid onsite 2 days/week or 8 days/month1 interview will be onsite (2 interview process)Databricks is a mustAWS or AzureMust be able to do code reviews of Python codeCreate designs and validate code"own" the architectureAs the Data Engineering Manager, you will be responsible for architecting, implementing, and optimizing end-to-end data solutions on...

  • Senior Data Engineer,

    8 hours ago

    Toronto, Ontario, Canada RBC Full time $120,000 - $180,000 per year

    Job DescriptionWhat is the opportunity?For a Senior Data Engineer to join our collaborative cyber security team, focusing on building scalable data solutions that directly enhance our application ecosystem. This role will work together with our application developers to integrate a robust data capability into our application to transform how our products...

  • 12+ Years Databricks Engineer

    7 days ago

    Toronto, Ontario, Canada Exdonuts Full time $150,000 - $200,000 per year

    12+ Years Databricks EngineerToronto, ON (4 days onsite)Must-Have:Hands-On development experience with Azure, Databricks, ADF, Pyspark, Python, SQL, BitBucket/GitHub;Release management process. SDLC and full unit testing understandingMust have 8+ years of solid experience in Data engineeringExperience working with ETL tools, in Agile format;Effective...