Principal Data Engineer (AWS/Python)
- Compensation
- Not disclosed
- Location
- London
- Work setup
- Not specified
- Seniority
- Principal
- Employment
- Full time
- Experience
- 8+ years
- Category
- Data engineering
- Posted
Skills
Nice to have
- Experience with Apache Spark for batch and streaming data processing at scale
- Experience with Apache Kafka, Confluent Platform, or AWS Kinesis for real-time streaming architectures
- Background in migrating Oracle or legacy database workloads to cloud-native architectures
- Familiarity with data mesh or data product principles
- Experience working as an ML engineer, or hands-on experience with ML platform integration (Amazon Bedrock, SageMaker, Databricks) and feature store design
- Knowledge of financial services, commodities, or energy market data, and/or experience working with timeseries data
- AWS Solutions Architect Professional or Data Analytics Specialty certification
Full description
Principal Data Engineer | London, UK | Department: Engineering | Hybrid (8 days per month onsite) | Full-time, Permanent | Industry: SaaS, Energy, Commodities & FinTech | Visa Sponsorship: Not available | REQ: 0031.About Us:
Founded in 1995, Zema Global Data Corporation empowers organizations to simplify complexity, reduce risk, and make faster, more confident decisions that drive measurable results. Over the past two years, Zema Global has accelerated its growth through strategic investment and acquisition to strengthen our global leadership. Together we’re helping our customers gain a Decisioning Advantage - one bold idea at a time. With a presence across global energy, commodity, and financial markets, Zema Global empowers customers to simplify complexity, reduce risk, and make faster, more confident decisions that drive measurable results. At Zema Global, we Think Big, Make It Happen, and Win as One. We thrive on collaboration, creativity, and respect, united by a shared drive to innovate and deliver meaningful impact for our customers and communities. If you’re inspired by solving complex challenges and contributing to a culture that values purpose and performance, we invite you to join us.
Position Overview:
We are looking for a Principal Data Engineer to define and own the technical architecture of our Data Surface programme, designing the end-to-end data platform on AWS including the Lakehouse layer (Apache Iceberg on S3), ingestion patterns, data modelling standards, and governance frameworks that the engineering team builds against. The architecture you define will also underpin our AI/ML ambitions: storing data in open source Apache Iceberg format so that AWS platforms such as Amazon Bedrock and SageMaker can consume it directly.
The Principal Data Engineer is a foundational role. The standards, patterns, and models you establish will shape how the platform is built and how our customers access commodities and energy market data for years to come. You will work alongside the UK engineering team and with global teams across the US, Canada, and India, with the autonomy to make architectural decisions on a programme that has senior sponsorship and genuine strategic weight.
What You'll Be Doing; Principal Data Engineer:
- As the Principal Data Engineer you will define and maintain the overall data architecture for the AWS Lakehouse platform, including Apache Iceberg table design, partitioning strategies, and storage layout on S3.
- Design data ingestion patterns from DynamoDB, Oracle, and streaming sources (Kinesis, Kafka) into Bronze-Silver-Gold transformation layers.
- Establish data modelling standards, naming conventions, and schema evolution policies across the platform.
- Architect data governance frameworks covering lineage tracking, access control, data quality rules, and metadata management using AWS Glue Data Catalog and related tooling.