Senior Data Engineer - AWS & Databricks (Public Sector)
- Compensation
- Not disclosed
- Location
- Singapore, Singapore, Singapore
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 6+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Degree or diploma in Computer Science, Software Engineering, Data Engineering, Information Systems or a related technical discipline.
- Typically 6–10 years of relevant data or software engineering experience, including production data pipeline design and operation.
- At least two years of experience working with cloud-based data platforms.
- Strong Python, PySpark and SQL skills, including distributed processing, complex transformations, query optimisation, debugging and automated testing.
- Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming data engineering.
- Practical AWS data engineering experience across relevant services such as S3, Glue, Lake Formation, IAM, KMS, Lambda, Step Functions, EventBridge, CloudWatch or Redshift.
- Experience designing lakehouse or data-lake architectures, layered data patterns, dimensional or domain data models and reusable datasets.
- Experience with incremental ingestion, change data capture, idempotency, late-arriving data, backfills, replay and recovery.
- Experience implementing data quality validation, schema enforcement, reconciliation, metadata and lineage in production pipelines.
- Familiarity with workflow orchestration, Git-based development, CI/CD, infrastructure as code such as Terraform and environment promotion.
- Experience implementing monitoring, operational dashboards, alerting, runbooks, incident diagnosis and service recovery.
- Working knowledge of cloud security, data classification, access controls, encryption and audit logging.
Nice to have
- Experience with Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming or Delta Sharing.
- Familiarity with Apache Airflow, dbt, Kafka, Amazon MSK/Kinesis or AWS DMS.
- Experience with data products, data contracts, semantic layers or data mesh-aligned operating models.
- Experience supporting feature engineering, ML training datasets, feature stores or MLOps pipelines.
- Familiarity with Singapore public-sector security, data classification and governed data-sharing requirements.
- Databricks or AWS data engineering certifications.
Full description
At Xtremax, we're looking for a Senior Data Engineer to design, build and operate dependable data pipelines and reusable data products on AWS and Databricks. You'll help establish the data foundations that support analytics, reporting and machine learning, ensuring that information remains reliable, well-governed and ready for production use.
You'll lead engineering work across data ingestion, transformation, modelling, quality validation, orchestration, lineage and performance optimisation. Using Python, PySpark, SQL, Apache Spark and Delta Lake, you'll build production-grade pipelines that handle complex data sources, incremental updates and operational recovery while maintaining clear data contracts and ownership.
This role combines hands-on engineering with technical leadership. You'll work with data owners, analysts, data scientists, architects and platform teams to translate business needs into maintainable data solutions, improve engineering standards and develop reusable patterns that make data services easier to operate and scale.
Responsibilities
- Lead the design and implementation of reliable batch and streaming pipelines across AWS and Databricks.
- Build lakehouse architectures using Delta Lake and layered Bronze, Silver and Gold data patterns.
- Develop reusable data models, curated datasets and data products with clear definitions, ownership and service expectations.
- Implement source integration and incremental processing for databases, files, APIs and event streams, including change data capture, replay and recovery.
- Establish data contracts, schema enforcement, automated quality checks, reconciliation and anomaly detection.
- Build and maintain workflows using Databricks Workflows, Lakeflow and relevant AWS orchestration services.
- Implement metadata, lineage, classification and access controls using Unity Catalog, AWS Glue Data Catalog, Lake Formation or approved platform capabilities.
- Develop monitoring, dashboards, alerts and operational runbooks for pipeline health, data freshness, quality, latency and downstream impact.
- Optimise Spark and SQL workloads, partitioning, clustering, caching and cluster configuration to improve performance and cost efficiency.
- Apply cloud security practices covering least-privilege access, encryption, secrets management, environment separation and audit logging.
- Review technical designs and code, establish engineering standards, mentor engineers and promote reusable components.
- Collaborate with data owners, analysts, data scientists and governance stakeholders to define data requirements and acceptance criteria.
- Maintain technical documentation and transfer operational knowledge to support sustainable internal ownership.