Astek
Senior Data Engineer – AWS / Databricks / PySpark
- Compensation
- Not disclosed
- Location
- Raffles Place, Central Region
- Work setup
- Onsite
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 2+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related discipline.
- 6–10 years of data/software engineering experience, including at least 2 years working with cloud-based data platforms.
- Strong hands-on experience with Python, PySpark, SQL, Apache Spark, Databricks and Delta Lake.
- Proficiency in AWS data services such as S3, Glue, Lake Formation, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis, IAM and KMS.
- Experience designing data lake/lakehouse architectures, data models and production-grade batch or streaming pipelines.
- Strong understanding of ETL/ELT, CDC, incremental ingestion, data quality, schema validation, metadata and lineage.
- Experience with workflow orchestration, Git, CI/CD, automated testing and infrastructure as code, preferably Terraform.
- Knowledge of data pipeline monitoring, observability, troubleshooting and performance optimisation.
- Familiarity with cloud security, encryption, access management and data governance.
- Proven ability to provide technical guidance, conduct code reviews and mentor engineers.
Nice to have
- Experience with Databricks Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles.
- Familiarity with Apache Airflow, dbt, Kafka, AWS MSK/Kinesis and AWS DMS.
- Knowledge of data contracts, data products, semantic layers, master/reference data and data mesh concepts.
- Experience supporting ML/AI data pipelines, feature engineering, feature stores or MLOps.
- Exposure to Singapore public-sector environments, including IM8, VAPT, data classification and cross-agency data governance.
Full description
Senior Data Engineer – AWS / Databricks / PySparkRole Overview
We are seeking an experienced Senior Data Engineer to design, develop and maintain scalable data pipelines and reusable data products on an AWS and Databricks-based data platform. The role involves technical leadership in data engineering, lakehouse architecture, data quality, governance and performance optimisation to support analytics, reporting and AI/ML workloads.
Key Responsibilities
- Design, develop and optimise scalable batch and streaming data pipelines using AWS, Databricks, Python, PySpark and SQL.
- Build and maintain lakehouse architectures using Delta Lake, including Bronze, Silver and Gold data layers, curated datasets and reusable data products.
- Implement data ingestion from databases, APIs, files and event streams, including incremental loading, Change Data Capture (CDC), backfills and recovery.
- Establish automated data quality checks, schema validation, reconciliation, metadata management and data lineage.
- Develop and automate data workflows using Databricks Workflows, Lakeflow and AWS services, incorporating CI/CD, automated testing and infrastructure as code.
- Implement data governance, access controls and security measures using Unity Catalog, AWS Glue Data Catalog, Lake Formation and IAM.
- Monitor pipeline performance, troubleshoot production issues and optimise Spark/SQL workloads for scalability, reliability and cost efficiency.
- Lead technical design and code reviews, establish engineering standards and mentor junior engineers.
- Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.
- Maintain technical documentation, data models, operational procedures and knowledge transfer materials.
Requirements
- Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related discipline.
- 6–10 years of data/software engineering experience, including at least 2 years working with cloud-based data platforms.
- Strong hands-on experience with Python, PySpark, SQL, Apache Spark, Databricks and Delta Lake.
- Proficiency in AWS data services such as S3, Glue, Lake Formation, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis, IAM and KMS.
- Experience designing data lake/lakehouse architectures, data models and production-grade batch or streaming pipelines.