Data Engineer

At UK Tote Group, we’re on a mission to reimagine the future of pool betting — building a modern, data‑driven betting experience for millions of racing fans. Our technology powers real‑time insights, supports responsible gaming, and helps us deliver trusted, customer‑first products across the UK and international markets. As a Data Engineer, you’ll play a key role in designing, building, and optimising the Databricks‑based Lakehouse that drives real‑time data, analytics, and reporting across the Tote. What You’ll Be Doing You’ll design, build, and optimise data pipelines using Databricks, Spark Structured Streaming, and Delta Live Tables to ensure data flows efficiently and reliably across the organisation. By applying the Medallion Architecture, you’ll develop robust Bronze, Silver, and Gold Delta tables that serve as the foundation for analytics, APIs, and decision‑making tools. You’ll integrate data from Kafka (MSK), AWS S3, and external APIs, ensuring seamless ingestion into the Lakehouse. You’ll work closely with BI teams to enable high‑performance Power BI dashboards through Databricks SQL Warehouses, making data more accessible and actionable across the business. You’ll ensure that all data is governed, discoverable, and secure under Unity Catalog, and you’ll contribute to continuous delivery by implementing and maintaining CI/CD pipelines for Databricks jobs, notebooks, and DLT workflows. Monitoring, tuning, and troubleshooting pipeline performance will also be part of your day‑to‑day, using Databricks metrics, CloudWatch, and AWS Cost Explorer to ensure optimal performance and efficiency. You’ll document data models, schemas, and lineage, maintaining a clear understanding of data flows and dependencies. Alongside Compliance and Technology colleagues, you’ll help ensure the platform remains fully compliant with GDPR and Gambling Commission regulations. Finally, you’ll champion best practices in data platform design, observability, and cost management, helping shape the future of the Tote’s data ecosystem. What We Are Looking For We’re looking for an experienced Data Engineer with proven expertise in building pipelines in Databricks and a strong grasp of Apache Spark, whether in PySpark or Scala, including Structured Streaming. You should have experience with Kafka (MSK) and real‑time data ingestion, as well as a deep understanding of Delta Lake, Delta Live Tables, and the Medallion Architecture. A strong AWS background is important, particularly with services such as S3, Glue, Lambda, Batch, and IAM. You’ll need to be proficient in Python and SQL for data engineering and analytics, and comfortable implementing CI/CD pipelines using tools such as GitHub Actions, Azure DevOps, or Jenkins. Solid experience with Git, version control, and Spark performance tuning will help you succeed in this role. Most importantly, you’ll bring a collaborative, proactive attitude and an ability to balance platform reliability with the pace of delivery. It would be an advantage if you have experience working with streaming architectures or data governance frameworks like Unity Catalog. Familiarity with Power BI, Looker, or Tableau is desirable, as is exposure to Databricks REST APIs, Airflow, or Databricks Workflows. Knowledge of infrastructure‑as‑code tools such as Terraform, AWS networking fundamentals, and cost management techniques using Photon and DBU monitoring will also be beneficial. You’ll be analytical, detail‑oriented, and motivated by solving complex data challenges. A self‑starter by nature, you take ownership of your work and enjoy designing end‑to‑end solutions that deliver measurable value. You’re a great communicator who can explain data concepts clearly to both technical and non‑technical colleagues, and you have a passion for automation, efficiency, and data quality. Most of all, you’re curious and committed to continuous learning within a fast‑evolving cloud data landscape. What’s in it for you? At the Tote you can expect a friendly working environment with a strong sense of teamwork and pride in what we do. Within this role you’ll develop a broad range of skills and experiences that can enhance your career at the Tote. Additionally, our company benefits package includes: Competitive Basic Salary Discretionary Bonus Scheme Company Shares Option Plan Contributory pension scheme Life insurance (4 x basic salary) Simply Health Cash Plan Holiday entitlement (33 days inclusive of bank holidays) Study Support and opportunity for progression and development Confidential 24/7 365 employee assistance helpline Agile and collaborative office environment with free parking, fruit, biscuits, and drinks Regular social events, charity events and volunteering opportunities. #J-18808-Ljbffr