Azure Databricks Developer
1 week ago
We are seeking an experienced Data Engineer with expertise in designing and implementing high-performance data ingestion pipelines. In this role, you will leverage Apache Spark and/or Azure Databricks to create efficient data pipelines that ingest data from multiple sources. You will also be responsible for delivering and presenting proofs of concept for key technology components to project stakeholders, and developing scalable, reusable frameworks for geospatial data sets. This position requires a strong technical background, the ability to think strategically, and a passion for innovation in data engineering.
Key Responsibilities
- Design and implement high-performance data ingestion pipelines using Apache Spark and Azure Databricks to efficiently process large-scale data from multiple sources.
- Develop scalable and reusable frameworks for ingesting and processing geospatial data sets.
- Deliver proofs of concept for key technology components to demonstrate their feasibility and functionality to project stakeholders.
- Collaborate with cross-functional teams, including data scientists, analysts, and product owners, to understand data requirements and integrate them into the pipeline architecture.
- Ensure data quality, integrity, and security across all stages of the pipeline.
- Optimize and fine-tune data processing workflows for maximum performance and cost efficiency.
- Continuously evaluate and integrate new data technologies and techniques to enhance the data engineering processes.
- Create comprehensive documentation of the data ingestion frameworks, processes, and solutions for internal and external stakeholders.
Required Qualifications
- Proven experience in data engineering, specifically with Apache Spark and/or Azure Databricks.
- Strong knowledge of data ingestion techniques and working with large-scale data sources.
- Experience with geospatial data formats and tools, as well as the ability to design systems for ingesting and processing geospatial data.
- Ability to design and present proofs of concept for technology solutions to stakeholders.
- Strong understanding of data architecture, ETL processes, and data integration.
- Experience with cloud technologies (especially Azure), and familiarity with distributed computing frameworks.
- Proficiency in programming languages such as Python, Scala, or Java for data processing.
- Strong communication skills, with the ability to collaborate effectively with both technical and non-technical stakeholders.
Preferred Qualifications
- Experience with Apache Hadoop, Kafka, or other distributed data processing frameworks.
- Familiarity with machine learning models and data pipelines in the context of big data processing.
- Knowledge of data governance and data security best practices.
- Experience in developing geospatial data systems or frameworks.
Certifications (if applicable)
Microsoft Certified: Azure Data Engineer or similar certifications are a plus.
#J-18808-Ljbffr
-
Azure Databricks Developer
3 weeks ago
Halifax, Canada Compunnel, Inc. Full timeWe are seeking an experienced Data Engineer with expertise in designing and implementing high-performance data ingestion pipelines. In this role, you will leverage Apache Spark and/or Azure Databricks to create efficient data pipelines that ingest data from multiple sources. You will also be responsible for delivering and presenting proofs of concept for key...
-
Azure Databricks Developer
2 weeks ago
Halifax, Canada Compunnel, Inc. Full timeWe are seeking an experienced Data Engineer with expertise in designing and implementing high-performance data ingestion pipelines. In this role, you will leverage Apache Spark and/or Azure Databricks to create efficient data pipelines that ingest data from multiple sources. You will also be responsible for delivering and presenting proofs of concept for key...
-
Halifax, Canada Compunnel, Inc. Full timeA technology solutions company in Halifax is seeking an experienced Data Engineer. You will design and implement high-performance data ingestion pipelines using Apache Spark and Azure Databricks. The role requires a strong technical background, expertise in geospatial data, and collaboration with cross-functional teams to ensure data quality and security....
-
Halifax, Canada Compunnel, Inc. Full timeA technology solutions company in Halifax is seeking an experienced Data Engineer. You will design and implement high-performance data ingestion pipelines using Apache Spark and Azure Databricks. The role requires a strong technical background, expertise in geospatial data, and collaboration with cross-functional teams to ensure data quality and security....
-
Halifax, Canada Compunnel, Inc. Full timeA technology solutions company in Halifax is seeking an experienced Data Engineer. You will design and implement high-performance data ingestion pipelines using Apache Spark and Azure Databricks. The role requires a strong technical background, expertise in geospatial data, and collaboration with cross-functional teams to ensure data quality and security....
-
SQL Database Developer
42 minutes ago
Halifax, Canada Compunnel, Inc. Full timeThe SQL Database Developer is responsible for designing, developing, and optimizing robust database solutions to support business processes and data-driven decision-making. The ideal candidate will demonstrate expertise in SQL, database architecture, performance tuning, and ETL processes. This role also emphasizes collaboration, analytical thinking, and the...
-
Data Integration Engineer
1 week ago
Halifax, Canada Compunnel, Inc. Full timeWe are seeking an experienced Data Integration Engineer to design, develop, and optimize scalable data solutions using Azure SQL and Azure Databricks. This role involves implementing data processing pipelines, ensuring data integrity and security, and collaborating with cross-functional teams to enhance performance and efficiency. Key Responsibilities 1....
-
SQL Database Developer
2 weeks ago
Halifax, Canada Compunnel, Inc. Full timeThe SQL Database Developer is responsible for designing, developing, and optimizing robust database solutions to support business processes and data-driven decision-making.The ideal candidate will demonstrate expertise in SQL, database architecture, performance tuning, and ETL processes.This role also emphasizes collaboration, analytical thinking, and the...
-
AI Model Developer
2 weeks ago
Halifax, Canada Compunnel, Inc. Full timeOverview We are seeking a skilled and innovative AI Model Developer to join our team. The ideal candidate will have hands-on experience with Large Language Models (LLMs) and a passion for building advanced AI solutions that enhance consumer experiences. This role involves developing, fine-tuning, and deploying AI models using cutting-edge technologies, with...
-
AI Model Developer
1 week ago
Halifax, Canada Compunnel, Inc. Full timeOverview We are seeking a skilled and innovative AI Model Developer to join our team. The ideal candidate will have hands-on experience with Large Language Models (LLMs) and a passion for building advanced AI solutions that enhance consumer experiences. This role involves developing, fine-tuning, and deploying AI models using cutting-edge technologies,...
