Data Engineer

Overview

Join our innovative team as a Data Engineer and become a vital driver of our data-driven initiatives In this dynamic role, you will design, develop, and optimize scalable data pipelines and architectures that empower our organization to harness the full potential of big data. You'll work with cutting-edge technologies such as AWS, Hadoop, Spark, and Azure Data Lake to build robust data solutions. Your expertise will enable seamless data integration, analysis, and modeling, fueling strategic decision-making across departments. This is an exciting opportunity for passionate professionals eager to make a tangible impact through data engineering excellence.

Responsibilities

• Develop, construct, test, and maintain scalable data pipelines using ETL tools like Talend, Informatica, and custom scripting with Bash or Shell Scripting to ensure reliable data flow across systems.
• Design and implement efficient data warehouse solutions utilizing SQL Server, Oracle, and modern cloud storage options such as Azure Data Lake and AWS S3.
• Collaborate with cross-functional teams to gather requirements and translate them into technical specifications for database design and data architecture enhancements.
• Optimize database performance through indexing, query tuning, and schema refinement to support high-volume Big Data processing with Apache Hive, Spark, and Hadoop ecosystems.
• Integrate diverse data sources including Linked Data and RESTful APIs to enrich datasets for comprehensive analysis.
• Support model training and analytics initiatives by preparing datasets and implementing machine learning workflows within Spark or Python environments.
• Maintain comprehensive documentation of data pipelines, architecture diagrams, and process workflows to ensure transparency and ease of maintenance.
• Participate actively in Agile development cycles, contributing to sprint planning, stand-ups, and continuous improvement efforts.
• Monitor system health and troubleshoot issues promptly to minimize downtime while ensuring security protocols are followed for sensitive information.

Experience

• Proven experience designing and deploying large-scale data pipelines using AWS cloud services such as S3, Lambda, or Glue.
• Strong proficiency in programming languages including Java and Python for data processing tasks.
• Extensive knowledge of Big Data technologies like Hadoop ecosystem components (HDFS, Hive), Spark (PySpark), and related tools.
• Hands-on experience with SQL Server, Oracle databases, and data warehouse architectures to support enterprise analytics needs.
• Familiarity with ETL tools such as Talend or Informatica along with scripting skills in Bash or Shell for automation purposes.
• Demonstrated ability to work with cloud platforms like Azure Data Lake or AWS for scalable storage solutions.
• Solid understanding of database design principles combined with analysis skills for optimizing complex queries and data models.
• Experience integrating Linked Data sources and developing RESTful APIs for seamless data exchange.
• Knowledge of analysis techniques related to model training and predictive analytics within a big data environment.
• Agile methodology experience with a focus on delivering high-quality solutions iteratively while adapting to evolving project requirements. Embark on this journey with us where your technical expertise fuels innovation We're dedicated to fostering an inclusive environment that values continuous learning—empowering you to grow your skills while making a meaningful impact through transformative data solutions.

Job Type: Full-time

Pay: $70,000.00-$120,000.00 per year

Work Location: In person