ETL QA Engineer with Databricks Expertise
1 week ago
Toronto ON, Toronto Census Division, ON; Ontario, Canada
Kumaran Systems
Full-time
Free with email or Google
Save this job and keep your search organized
Create a free account to save jobs, create alerts and return to this listing from your dashboard.
Free with email or Google
By continuing, you agree to our Terms & Privacy Policy.
Elevate data quality as an ETL QA Engineer specializing in Databricks and PySpark. This position emphasizes AI-assisted development and efficient data pipeline validation. We are seeking a highly skilled ETL QA Engineer with extensive experience in Databricks and expert-level PySpark capabilities. You'll focus on ensuring the integrity of large-scale data pipelines and leverage tools such as GitHub Copilot and AI agents to increase testing efficiency. Your role will also include contributing to AI-driven knowledge bases and fostering a robust testing culture.
Key Responsibilities
• Validate large-scale data pipelines for quality assurance
• Utilize PySpark for transformations and optimization techniques
• Implement CI/CD using GitHub Copilot and actions
• Construct effective prompts for testing with AI tools
• Document complex data validation processes clearly
Requirements:
• Expertise in Databricks notebooks, jobs, and Delta Lake
• Proven proficiency in automated testing frameworks
• Strong understanding of ETL/ELT concepts and data warehousing
• Experience with agent-based architectures in QA
• Detail-oriented approach to problem-solving Drive data excellence through innovative QA practices and leverage AI tools to enhance efficiency. #J-18808-Ljbffr
Key Responsibilities
• Validate large-scale data pipelines for quality assurance
• Utilize PySpark for transformations and optimization techniques
• Implement CI/CD using GitHub Copilot and actions
• Construct effective prompts for testing with AI tools
• Document complex data validation processes clearly
Requirements:
• Expertise in Databricks notebooks, jobs, and Delta Lake
• Proven proficiency in automated testing frameworks
• Strong understanding of ETL/ELT concepts and data warehousing
• Experience with agent-based architectures in QA
• Detail-oriented approach to problem-solving Drive data excellence through innovative QA practices and leverage AI tools to enhance efficiency. #J-18808-Ljbffr