Sr. Data Mesh Architect

1 week ago

Canada Amicon Hub Services Full time

About QSpark Group :-

QSpark Group is a leading provider of innovative solutions in the field of data modernization, analytics and AI. We specialize in leveraging cutting-edge technologies to help businesses build and manage scalable data lakes and data lakehouses, enabling them to harness the full potential of their data assets for strategic decision-making.


Job Description

As a Data Lake/Data Lakehouse Architect at QSpark Group, you will be responsible for designing, implementing, and optimizing our next-generation data lake architecture. You will lead the development of scalable data platforms using Apache Spark and modern table formats like Apache Iceberg, ensuring high performance, reliability, and cost-effectiveness of our data infrastructure.


Key Responsibilities

1. Data Lake Architecture: Design and implement modern data lake architectures using best practices for data organization, partitioning, and storage optimization.

2. Spark Development: Lead the development of data processing applications using PySpark, optimizing for performance and resource utilization across large-scale Spark clusters.

3. Table Format Implementation: Design and implement data storage strategies using Apache Iceberg, enabling ACID transactions, time travel, and schema evolution capabilities.

4. Data Pipeline Development: Create robust and scalable data pipelines for batch and streaming workflows, implementing proper error handling and data quality checks.

5. Performance Optimization: Tune Spark applications and cluster configurations for optimal performance, implementing partitioning strategies and query optimization techniques.

6. Data Governance: Establish and maintain data governance frameworks specific to data lake environments, including access controls, audit logging, and compliance measures.

7. Infrastructure as Code: Develop and maintain infrastructure as code for automated deployment of data lake components and Spark clusters.

8. Cross-functional Leadership: Collaborate with data engineers, data scientists, and business stakeholders to ensure the data lake architecture meets both technical and business requirements.


Requirements

- Bachelor's or Master's degree in Computer Science, Data Engineering, or related field.

- 12+ years of experience in data engineering, with at least 3 years focused on data lake architecture and implementations.

- Deep expertise in Apache Spark and PySpark, including:

 - Complex ETL development

 - Performance tuning and optimization

 - Spark SQL and DataFrame APIs

 - Spark cluster management and monitoring

- Strong experience with Apache Iceberg, including:

 - Table management and optimization

 - Schema evolution

 - Time travel operations

 - Partition evolution

- Proficiency in Python programming and software engineering best practices.

- Experience with cloud platforms (AWS, Azure, or GCP) and their respective data lake services.

- Strong understanding of data modeling concepts for both structured and unstructured data.

- Experience with version control systems (Git) and CI/CD pipelines.


Preferred Qualifications

- Experience with other modern table formats (Delta Lake, Apache Hudi).

- Knowledge of data catalog solutions (AWS Glue, Azure Data Catalog).

- Familiarity with data streaming technologies (Kafka, Spark Streaming).

- Experience with container orchestration platforms (Kubernetes).

- Certification in relevant cloud platforms or data technologies.

- Contributions to open-source projects in the data space.


Technical Skills

- Languages: Python, SQL, Scala (optional)

- Frameworks: Apache Spark, PySpark

- Table Formats: Apache Iceberg

- Infrastructure: AWS EMR, Databricks, or equivalent

- Version Control: Git, GitHub/GitLab

- CI/CD: Jenkins, GitHub Actions, Notebook+Git or equivalent

- Monitoring: Grafana, Prometheus, or equivalent


Why Join QSpark Group

- Opportunity to work with cutting-edge data lake technologies and shape the future of data architecture.

- Competitive salary and comprehensive benefits package, including health insurance.

- Professional development support, including conference attendance and certification programs.

- Remote-friendly work environment with flexible scheduling options.

- Collaborative culture that values innovation, technical excellence, and continuous learning.

- Regular opportunities to contribute to open-source projects and technical community initiatives.

  • Sr. Data Mesh Architect

    1 week ago

    Canada, CA Amicon Hub Services Full time

    About QSpark Group :- QSpark Group is a leading provider of innovative solutions in the field of data modernization, analytics and AI. We specialize in leveraging cutting-edge technologies to help businesses build and manage scalable data lakes and data lakehouses, enabling them to harness the full potential of their data assets for strategic...

  • Sr. SAP PP/QM Functional

    1 week ago

    Canada NTT DATA North America Full time

    Sr. SAP PP/QM Functional Consultant with Data Migration.Position OverviewOur SAP Implementation PP/QM Functional Consultant will lead others through the implementation of SAP. Duties include but may not be limited to:Educate local teams on SAP Production Planning/Quality Management functionality, and SAP, in generalRefine the Plan to Manufacture process...

  • Sr. SAP PP/QM Functional

    1 week ago

    Canada NTT DATA North America Full time

    Sr. SAP PP/QM Functional Consultant with Data Migration.Position OverviewOur SAP Implementation PP/QM Functional Consultant will lead others through the implementation of SAP. Duties include but may not be limited to:Educate local teams on SAP Production Planning/Quality Management functionality, and SAP, in generalRefine the Plan to Manufacture process...

  • Sr. SAP PP/QM Functional

    1 week ago

    Canada, CA NTT DATA North America Full time

    Sr. SAP PP/QM Functional Consultant with Data Migration.Position OverviewOur SAP Implementation PP/QM Functional Consultant will lead others through the implementation of SAP. Duties include but may not be limited to:Educate local teams on SAP Production Planning/Quality Management functionality, and SAP, in generalRefine the Plan to Manufacture process...

  • Data Solutions Architect

    5 days ago

    Canada IMCS Group Full time

    Job Title: Data Solutions ArchitectAbout the Role:We are seeking an experienced Data Solutions Architect to join our team at IMCS Group in Edmonton, AB. This is a 6-month contract position with possible extension.About the Job:The proposed resource must meet or exceed one of the following criteria:University graduation in business, management, or a related...

  • Cloud Data Architect

    2 days ago

    Canada hireVouch Full time

    We are seeking a highly skilled Cloud Data Architect to join our team at hireVouch.As a Cloud Data Architect, you will play a critical role in designing and implementing data environments to support reporting and analytics for our clients.The ideal candidate will have experience with AWS infrastructure and technologies, including DMS, Glue, and Step...

  • Data Architect

    5 days ago

    Canada IMCS Group Full time

    Data Architect – Intermediate6 Months Contract with possible extensionRemote / Edmonton, ABThe proposed resource must meet or exceed one of the following:University graduation in business, management or a related discipline and 2-yr Data Architect (DA) experience.OR2-yr diploma in business, management or a related discipline and 4-yr DA experience.OR1-yr...

  • Data Architect

    4 days ago

    Canada IMCS Group Full time

    Data Architect – Intermediate6 Months Contract with possible extensionRemote / Edmonton, ABThe proposed resource must meet or exceed one of the following:University graduation in business, management or a related discipline and 2-yr Data Architect (DA) experience.OR2-yr diploma in business, management or a related discipline and 4-yr DA experience.OR1-yr...

  • Data Architect

    5 days ago

    Canada IMCS Group Full time

    Data Architect – Intermediate 6 Months Contract with possible extension Remote / Edmonton, AB The proposed resource must meet or exceed one of the following: University graduation in business, management or a related discipline and 2-yr Data Architect (DA) experience. OR 2-yr diploma in business, management or a related discipline and 4-yr DA...

  • Data Architect

    5 days ago

    Canada, CA IMCS Group Full time

    Data Architect – Intermediate6 Months Contract with possible extensionRemote / Edmonton, ABThe proposed resource must meet or exceed one of the following:University graduation in business, management or a related discipline and 2-yr Data Architect (DA) experience.OR2-yr diploma in business, management or a related discipline and 4-yr DA experience.OR1-yr...

  • Application Architect

    2 weeks ago

    Greater Toronto Area, Canada, Ontario PrecisionERP Incorporated Full time

    PrecisionERP/IT is recruiting for a Sr. Applications Architect for a HYBRID opportunity (3 days ONSITE) with our client based in either Winnipeg or Toronto offices.The Sr. Applications Architect with include the responsibility for the overall architecture, development, and integration of multi-platform information technology solutions within projects, as...

  • Senior Insurance Data Architect

    1 week ago

    Canada Liztek Full time

    At Liztek, we're seeking a seasoned Senior Insurance Data Architect to join our team. This is an exciting opportunity to work with cutting-edge technology and drive innovation in the insurance industry.About the RoleWe're looking for a highly skilled and experienced data professional who can design, develop, and maintain complex data architectures in...

  • Data Architect Lead for Cloud Data Platforms

    2 days ago

    Canada PrecisionERP Incorporated Full time

    Job OverviewPrecisionERP Incorporated is seeking an experienced Senior Data Architect to lead our cloud data platform initiatives.About the RoleThis full-time, 35-hour per week position offers a hybrid work arrangement with 2 days on-site in downtown Montreal and 3 days remotely.Estimated Salary: $120,000 - $160,000 per year, based on industry standards and...

  • Cloud Data Architect

    2 days ago

    Canada Procom Full time

    Job Overview:We are seeking a skilled Cloud Data Architect to join our Procom team.About the Role:The Cloud Data Architect will be responsible for designing and implementing comprehensive Azure solutions, considering infrastructure, applications, data, security, and ETL development. This includes Azure administration skills and experience with Azure Data...

  • Data Architect Lead

    2 days ago

    Canada Resonaite Full time

    About the RoleWe are seeking an experienced Data Architect to lead our end-to-end data architecture efforts for a critical system rebuild project. As a key member of our team, you will be responsible for designing and implementing scalable, future-ready data integration solutions.The successful candidate will have expertise in SAP and a proven track record...

  • Data Engineer

    7 days ago

    Canada Liztek Full time

    Job Title:Data Engineer - Insurance Data ArchitectCompany Overview:Liztek is a cutting-edge technology firm dedicated to revolutionizing the insurance industry through innovative data solutions.Salary:$120,000 - $180,000 per year, depending on experience and qualifications.About the Job:We are seeking an experienced Data Engineer with expertise in Snowflake...

  • Data Engineer

    1 week ago

    Canada, CA technology& Full time

    Job Title: Data EngineerExperience Range: 8-14 yearsLocation: Canada (100% Remote, must be based within Canada)Job Type: ContractDuration: 6+ MonthsRate: CAD $60/hr - $65/hr (all-inclusive)Work Authorization: Open to all candidates legally authorized to work in Canada (No third-party candidates)Start Date: ASAPJob DescriptionAs a Data Engineer, you will play...

  • Chief Data Architect

    2 days ago

    Canada Electric Mind inc. Full time

    Company OverviewElectric Mind inc. is a dynamic organization that fosters innovation and growth in the field of data management and analytics.Job SummaryWe are seeking an experienced Chief Data Architect to join our team, contributing to the design and implementation of high-quality data pipelines and architectures.ResponsibilitiesDevelop and maintain...

  • Senior Data Architect

    2 days ago

    Canada LeverageTek Staffing Solutions Full time

    About the RoleLeverageTek Staffing Solutions is seeking a highly skilled Senior Data Architect to lead our data architecture initiatives. This is a 6-month contract opportunity with a remote work arrangement and occasional travel to Montreal (5-10% travel requirement).Key ResponsibilitiesDevelop and implement data ETL procedures, standards, and metrics to...

  • Lead Data Architect

    4 months ago

    Canada Hillsdale Investment Management Inc. Full time

    LEAD DATA ARCHITECTHillsdale Investment Management Inc. (Hillsdale) is an independent, employee-controlled, and client-aligned investment boutique managing investments on behalf of a select group of sophisticated institutional and private wealth investors. We engage in disciplined active investment management, specializing in equities, alternatives, ESG, and...