Lead/Principle Data Engineer

Data Engineer
Clearance Required: Active SC Clearance (Mandatory)

The Opportunity

We’re looking for a talented Data Engineer to join a growing team delivering secure, large-scale data solutions for government and public sector customers.

This is an exciting opportunity to work on complex data platforms that support critical decision-making and operational outcomes. You’ll play a key role in building and enhancing secure batch and real-time data processing capabilities, working with modern cloud technologies and industry-leading open-source tools.

As part of a collaborative Agile team, you’ll help design, develop, and optimise data pipelines, ensuring data is accurately processed, transformed, and delivered to downstream consumers. This role offers exposure to cutting-edge cloud and data technologies alongside the opportunity to work on meaningful projects within highly secure environments.

Please note: Active SC Clearance is an essential requirement for this position. Due to the nature of the work, candidates must already hold current and active SC Clearance. Unfortunately, sponsorship or clearance applications cannot be supported for this role.

Key Responsibilities

• Develop, test, and maintain robust data pipelines that meet both functional and non-functional business requirements.
• Design, build, and optimise data processing solutions to support batch and real-time analytics capabilities.
• Contribute to logical and physical database design activities across a range of data platforms.
• Expand and enhance platform capabilities, enabling the business to solve new and evolving data challenges.
• Analyse complex datasets and identify opportunities to improve data processing, quality, and performance.
• Ensure data models align with established data architecture standards, maintaining consistency across entities, relationships, and definitions.
• Perform root cause analysis on data-related issues, identifying and implementing effective solutions.
• Collaborate with engineers, architects, and stakeholders to deliver scalable and reliable data solutions.
• Support continuous improvement initiatives, helping to drive best practices across data engineering and platform development.
• Produce clear technical documentation covering data pipelines, processes, and system enhancements.

Skills & Experience

Essential

• Experience developing and maintaining data pipelines using modern data processing technologies and frameworks.
• Knowledge of Agile delivery methodologies and experience working within fast-paced development environments.
• Strong understanding of data modelling, data structures, and their advantages and limitations across different use cases.
• Experience working with public cloud platforms, particularly AWS, including services such as EC2, RDS, Lambda, Athena, and Glue.
• Experience configuring and optimising Relational and NoSQL databases, including query tuning and performance optimisation.
• Proficiency in programming and scripting languages such as Python.
• Experience applying Test-Driven Development (TDD) methodologies and associated testing frameworks.
• Strong analytical and problem-solving skills with the ability to work through complex data challenges.
• Excellent communication skills and the ability to work effectively within multidisciplinary teams.
• Bachelor’s degree or equivalent commercial experience in a relevant discipline.

Desirable

• Experience with CI/CD tooling such as Jenkins or Drone.
• Knowledge of configuration management tools including Ansible and Packer.
• Experience using Terraform for Infrastructure-as-Code deployments.
• Understanding of both Linux and Windows environments.
• Exposure to modern cloud-native architectures and DevOps practices.
• Experience working within government, defence, or other highly regulated environments.

What You’ll Bring

• A passion for data engineering and solving complex technical problems.
• A strong attention to detail and commitment to delivering high-quality solutions.
• A collaborative mindset and the ability to work effectively within Agile teams.
• A proactive approach to learning new technologies and expanding your technical skillset.
• The ability to translate business requirements into scalable and efficient data solutions.
• A desire to contribute to projects that deliver meaningful impact within secure and mission-critical environments.

Security Clearance – Mandatory Requirement

Candidates must hold current and active SC (Security Check) Clearance at the time of application.

Applications from candidates who are eligible for clearance, willing to undergo clearance, or whose clearance has expired cannot be considered for this position.

This is an excellent opportunity to join a high-performing engineering team, working on innovative cloud and data platforms that support critical government programmes while developing your expertise with modern technologies and large-scale data solutions.

Reasonable Adjustments:

Respect and equality are core values to us. We are proud of the diverse and inclusive community we have built, and we welcome applications from people of all backgrounds and perspectives. Our success is driven by our people, united by the spirit of partnership to deliver the best resourcing solutions for our clients.

If you need any help or adjustments during the recruitment process for any reason, please let us know when you apply or talk to the recruiters directly so we can support you.