Senior Data Engineer AWS / Databricks / PySpark
Senior Data Engineer designing, building and optimizing scalable batch and streaming data pipelines and lakehouse data products on an AWS and Databricks platform using Python, PySpark and SQL. Covers data quality, governance (Unity Catalog, Lake Formation), CI/CD/IaC and technical leadership with mentoring of junior engineers.
- Compensation
- Not disclosed
- Location
- Singapore
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 2+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or related discipline
- 6-10 years of data/software engineering experience, including at least 2 years with cloud-based data platforms
- Strong hands-on experience with Python, PySpark, SQL, Apache Spark, Databricks and Delta Lake
- Proficiency in AWS data services (S3, Glue, Lake Formation, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis, IAM, KMS)
- Experience designing data lake/lakehouse architectures, data models and production-grade batch or streaming pipelines
- Strong understanding of ETL/ELT, CDC, incremental ingestion, data quality, schema validation, metadata and lineage
- Experience with workflow orchestration, Git, CI/CD, automated testing and infrastructure as code, preferably Terraform
- Knowledge of data pipeline monitoring, observability, troubleshooting and performance optimisation
- Familiarity with cloud security, encryption, access management and data governance
- Proven ability to provide technical guidance, conduct code reviews and mentor engineers
Nice to have
- Experience with Databricks Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles
- Familiarity with Apache Airflow, dbt, Kafka, AWS MSK/Kinesis and AWS DMS
- Knowledge of data contracts, data products, semantic layers, master/reference data and data mesh concepts
- Experience supporting ML/AI data pipelines, feature engineering, feature stores or MLOps
- Exposure to Singapore public-sector environments (IM8, VAPT, data classification, cross-agency data governance)
Full description
Senior Data Engineer - AWS / Databricks / PySpark
Role Overview
We are seeking an experienced
Senior Data Engineer
to design, develop and maintain scalable data pipelines and reusable data products on an AWS and Databricks-based data platform. The role involves technical leadership in data engineering, lakehouse architecture, data quality, governance and performance optimisation to support analytics, reporting and AI/ML workloads.
Key Responsibilities
Design, develop and optimise scalable batch and streaming data pipelines using
AWS, Databricks, Python, PySpark and SQL .
Build and maintain lakehouse architectures using
Delta Lake , including Bronze, Silver and Gold data layers, curated datasets and reusable data products.
Implement data ingestion from databases, APIs, files and event streams, including incremental loading, Change Data Capture (CDC), backfills and recovery.
Establish automated data quality checks, schema validation, reconciliation, metadata management and data lineage.
Develop and automate data workflows using
Databricks Workflows, Lakeflow and AWS services , incorporating CI/CD, automated testing and infrastructure as code.
Implement data governance, access controls and security measures using
Unity Catalog, AWS Glue Data Catalog, Lake Formation and IAM .
Monitor pipeline performance, troubleshoot production issues and optimise Spark/SQL workloads for scalability, reliability and cost efficiency.
Lead technical design and code reviews, establish engineering standards and mentor junior engineers.
Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.
Maintain technical documentation, data models, operational procedures and knowledge transfer materials.
Requirements
Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related discipline.
6-10 years of data/software engineering experience , including at least 2 years working with cloud-based data platforms.
Strong hands-on experience with
Python, PySpark, SQL, Apache Spark, Databricks and Delta Lake .
Proficiency in AWS data services such as
S3, Glue, Lake Formation, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis, IAM and KMS .