Data Engineer
Data Engineer at the Hong Kong Jockey Club designing, building, and operating large-scale ETL/ELT pipelines with PySpark and SQL on AWS, managing a cloud data lake with medallion architecture, real-time streaming, and DevOps/DataOps practices to support the Club's digital transformation.
- Compensation
- Not disclosed
- Location
- Hong Kong, Hong Kong
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- 3+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Proven expertise in PySpark for large-scale data processing at terabyte to petabyte scale
- Strong experience building and managing production-level ETL workflows and frameworks
- Hands-on experience with AWS or other cloud data analytics services
- Hands-on experience with data visualisation tools such as Power BI or Tableau
- 5-8 years of hands-on experience in data engineering designing, building, and maintaining scalable cloud ETL/ELT pipelines
- 5-8 years of production experience using PySpark and SQL for large-scale data processing
- 3+ years of experience with end-to-end data pipeline lifecycle management including monitoring, observability, and production support
- 3+ years of experience architecting, implementing, and managing cloud data platforms on AWS
- Experience with data governance, data security best practices, and compliance requirements
- Experience with real-time data streaming technologies such as Kafka or Solace
- Knowledge of Medallion architecture principles, design patterns, and data modelling
- Experience with infrastructure-as-code using Terraform or CloudFormation
- Experience with DevOps/DataOps practices including CI/CD, automated testing, and pipeline monitoring
- Background working with on-premises data platforms and migration to the cloud
Nice to have
- AWS certifications such as Data Analytics Specialty, Solutions Architect Associate/Professional
Full description
About the Role
The role contributes to shaping the future of sports and entertainment at the Club through a strategic mission to transform digital capabilities. It sits within the AOP programme, which aims to accelerate platform, product, people, performance, and potential through innovation and delivery excellence. The work spans product, technology, operations, delivery, and change, creating next-generation experiences and strengthening the capabilities that support the Club's future growth. The team is diverse and passionate about solving complex challenges, delivering meaningful impact, and creating value for customers, members, the Club, and the wider community.
Key Responsibilities
- Design, develop, test, and operationalise reliable and scalable ETL/ELT data pipelines using PySpark and SQL on the AWS platform.
- Implement and manage a cloud data lake and orchestrate data flows following the medallion architecture.
- Optimise data pipelines and storage solutions for performance, cost, and reliability while handling huge volumes of data.
- Communicate with business users and product owners as required.
Required Experience & Qualifications
- Proven expertise in PySpark for large-scale data processing at terabyte to petabyte scale.
- Strong experience building and managing production‑level ETL workflows and frameworks.
- Hands‑on experience with AWS or other cloud data analytics services.
- Hands‑on experience with data visualisation tools such as Power BI or Tableau.
- 5–8 years of hands‑on experience in data engineering, focusing on designing, building, and maintaining scalable cloud ETL/ELT pipelines.
- 5–8 years of production experience using PySpark and SQL for large-scale data processing.
- 3+ years of experience with end‑to‑end data pipeline lifecycle management, including monitoring, observability, and production support.
- 3+ years of experience architecting, implementing, and managing cloud data platforms on AWS.
- Experience with data governance and data security best practices and compliance requirements.
- Experience with real‑time data streaming technologies such as Kafka or Solace.
- Knowledge of Medallion architecture principles, design patterns, and data modelling.
- Experience with infrastructure‑as‑code using Terraform or CloudFormation.
- Experience with DevOps/DataOps practices including CI/CD, automated testing, and pipeline monitoring.
- Background working with on‑premises data platforms and migration to the cloud.
- AWS certifications such as Data Analytics Speciality, Solutions Architect Associate/Professional.
Salary
Competitive salary