Lead/Principle Data Engineer

Overview

You will design, build, and optimise secure data pipelines for large-scale government data platforms. In an Agile, cross-functional team, you’ll deliver batch and real-time processing solutions that support critical decision-making. This role emphasises secure environments, modern cloud tech, and open-source tools to drive reliable data delivery and platform evolution. You’ll work closely with engineers and stakeholders to ensure data quality, performance, and compliance, contributing to mission-critical outcomes.

Responsibilities

• Develop, test, and maintain robust data pipelines meeting business requirements
• Design and optimise batch and real-time data processing solutions
• Contribute to logical/physical database design across data platforms
• Enhance platform capabilities to tackle new data challenges
• Analyse complex datasets to improve processing, quality, and performance
• Ensure data models align with architecture standards and definitions
• Perform root cause analysis on data issues and implement solutions
• Collaborate with engineers, architects, and stakeholders to deliver scalable data solutions
• Support continuous improvement and best practices in data engineering and platform development
• Produce technical documentation for pipelines and system enhancements

Key requirements

• Experience building and maintaining data pipelines with modern data processing frameworks
• Experience with Agile delivery in fast-paced environments
• Strong data modelling knowledge and understanding of data structures
• Experience with AWS (EC2, RDS, Lambda, Athena, Glue) applications
• Relational and NoSQL database experience with query tuning and optimization
• Proficiency in Python
• Experience applying Test-Driven Development (TDD)
• Strong analytical and problem-solving abilities
• Excellent communication and cross-functional collaboration
• Bachelor’s degree or equivalent experience

• Collaborative mindset
• Strong communication
• Analytical thinking
• AWS services (EC2, RDS, Lambda, Athena, Glue)
• Data pipelines and processing frameworks
• Data modelling and database design