Senior Data Engineer

Job Title: Senior Data Engineer - Platform Engineering Company Overview: Atreides helps organizations transform large and complex multi-modal datasets into information-rich geo-spatial data subscriptions that can be used across a wide spectrum of use cases. Currently, Atreides focuses on providing high-fidelity data solutions to enable customers to derive insights quickly. Atreides transforms the chaos of petabyte-scale, all-domain data—land, air, sea, space, and cyber—into real-time operational clarity. We are a fast-moving, high-performance international scale company. We trust our team with autonomy, believing it leads to better results and job satisfaction. With a mission-driven mindset and entrepreneurial spirit, we are building something new and helping unlock the power of massive-scale data to make the world safer, stronger, and more prosperous. Team Overview: We are a passionate team of technologists, data scientists, and analysts with backgrounds in operational intelligence, law enforcement, large multinationals, and cybersecurity operations. We obsess about designing products that will change the way global companies, governments and nonprofits protect themselves from external threats and global adversaries. Position Overview: As a Senior Data Engineer on the Platform Engineering team, you will lead the architecture and implementation of high-scale data systems that process vast quantities of geospatial and sensor data. Your work will involve building fault-tolerant, high-throughput pipelines, optimizing distributed data workflows, and enabling robust APIs and data access mechanisms that support advanced analytics and visualization. You’ll be responsible for setting engineering standards, guiding junior developers, and collaborating with data scientists, analysts, and product teams to shape the long‑term roadmap of Atreides’ platform capabilities. This role combines technical leadership with deep hands‑on development in a high-performance environment. Team Principles: At Atreides, we believe that teams work best when they: Remain curious and passionate in all aspects of our work Promote clear, direct, and transparent communication Embrace the 'measure twice, cut once' philosophy Value and encourage diverse ideas and technologies Lead with empathy in all interactions Responsibilities: Architect scalable systems for ingesting, processing, and querying high-volume datasets. Lead development of core data infrastructure components, including scalable data pipelines. Design, implement, and maintain robust orchestration frameworks (e.g., Apache Airflow, Dagster, Prefect) to schedule, monitor, and manage large-scale data workflows. Develop and maintain self‑service platform SDKs, libraries, and tooling to empower other engineering and analytics teams to build, deploy, and monitor pipelines independently. Optimize the performance, cost‑efficiency, and reliability of platform services. Ensure high availability, observability, and compliance for platform components. Implement secure data handling practices and fault‑tolerant systems. Mentor junior engineers and provide code reviews, guidance, and architectural leadership. Collaborate with analytics and customer‑facing teams to align infrastructure with use‑case requirements. Desired Qualifications: 5+ years of experience building and maintaining production‑grade data systems. Deep expertise in distributed processing frameworks (e.g., Spark, Flink, Kafka) and cloud architecture. Advanced proficiency in Python and SQL (especially Spark SQL). Experience developing internal platform tooling or SDKs that enable self‑service data and workflow management. Experience with geospatial data processing, formats, and services (e.g., Apache Sedona). Strong understanding of CI/CD workflows, observability tools, and platform reliability engineering. Compensation and Benefits: Competitive salary Comprehensive health, dental, and vision insurance plans Flexible hybrid work environment Additional benefits like flexible hours, work travel opportunities, competitive vacation time and parental leave While meeting all of these criteria would be ideal, we understand that some candidates may meet most, but not all. If you’re passionate, curious and ready to "work smart and get things done," we’d love to hear from you. #J-18808-Ljbffr