Data Engineer

3 hours ago

ca ca Spait Infotech Full-time
  • Design, develop, and maintain scalable data pipelines.
  • Build and manage ETL/ELT workflows.
  • Extract, transform, and load data from multiple sources.
  • Develop data processing solutions using Python, SQL, and PySpark.
  • Work with structured and unstructured data.
  • Design and maintain data warehouses and data lakes.
  • Develop automated workflows using Apache Airflow.
  • Integrate data from APIs, databases, and external systems.
  • Ensure data quality, accuracy, and consistency.
  • Monitor, troubleshoot, and optimize data pipelines.
  • Implement data validation, governance, and security.
  • Collaborate with Data Scientists, Data Analysts, and BI teams.

Preferred Skills

  • Strong knowledge of Python, SQL, and PySpark.
  • Experience with Apache Spark and Hadoop.
  • Knowledge of ETL/ELT processes.
  • Hands-on experience with Apache Airflow.
  • Knowledge of Apache Kafka and real-time data processing.
  • Experience with MySQL, PostgreSQL, and MongoDB.
  • Knowledge of Snowflake, Databricks, Redshift, or BigQuery.
  • Experience with AWS, Azure, or GCP cloud platforms.
  • Knowledge of AWS Glue and Azure Data Factory.
  • Understanding of Data Warehousing and Data Modelling.
  • Experience with Data Lakes and Lakehouse architecture.
  • Knowledge of Git, GitHub, and CI/CD pipelines.
  • Familiarity with Docker and Kubernetes.
  • Knowledge of Linux and Shell scripting.
  • Strong problem-solving and analytical skills.