Key Responsibilities

Design, develop and maintain scalable batch and streaming data pipelines across AWS and Databricks.
Build and manage lakehouse architectures using Apache Spark, PySpark and Delta Lake, including Bronze, Silver and Gold data layers.
Integrate data from databases, APIs, files and event streams using incremental ingestion and change data capture.
Implement data quality checks, schema validation, reconciliation, metadata management and data lineage.
Develop automated workflows and CI/CD pipelines using Databricks Workflows, AWS services, Git and infrastructure-as-code tools.
Monitor pipeline performance, troubleshoot issues and implement logging, alerts and recovery procedures.
Optimise Spark and SQL workloads for performance, reliability and cloud cost efficiency.
Implement data security, access controls, encryption and governance in line with agency requirements.
Establish engineering standards, review code and architecture, and mentor junior engineers.
Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.
Maintain technical documentation, operating procedures and knowledge-transfer materials.

Requirements & Qualifications

Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related field.
6–10 years of relevant data or software engineering experience, including at least 2 years working with cloud-based data platforms.
Strong hands-on experience in Python, PySpark, SQL, Apache Spark and Delta Lake.
Proven experience building and operating production data pipelines using Databricks and AWS.
Good knowledge of AWS services such as S3, Glue, Lake Formation, IAM, Lambda, Redshift and CloudWatch.
Experience with lakehouse architecture, data modelling, incremental processing, change data capture and data quality management.
Familiarity with workflow orchestration, CI/CD, Git and infrastructure-as-code tools such as Terraform.
Experience with pipeline monitoring, troubleshooting, performance optimisation and cloud security.
Ability to lead technical design, conduct code reviews, establish engineering standards and mentor engineers.
Strong problem-solving, communication and stakeholder management skills.

Preferred Skills

Experience with Unity Catalog, Databricks Workflows, Lakeflow, Delta Live Tables or Structured Streaming.
Familiarity with Airflow, dbt, Kafka, AWS DMS, MSK or Kinesis.
Experience building data products, data contracts, feature engineering or ML data pipelines.
Knowledge of Singapore public-sector data governance, security and compliance requirements.

Key Responsibilities

Design, develop and maintain scalable batch and streaming data pipelines across AWS and Databricks.
Build and manage lakehouse architectures using Apache Spark, PySpark and Delta Lake, including Bronze, Silver and Gold data layers.
Integrate data from databases, APIs, files and event streams using incremental ingestion and change data capture.
Implement data quality checks, schema validation, reconciliation, metadata management and data lineage.
Develop automated workflows and CI/CD pipelines using Databricks Workflows, AWS services, Git and infrastructure-as-code tools.
Monitor pipeline performance, troubleshoot issues and implement logging, alerts and recovery procedures.
Optimise Spark and SQL workloads for performance, reliability and cloud cost efficiency.
Implement data security, access controls, encryption and governance in line with agency requirements.
Establish engineering standards, review code and architecture, and mentor junior engineers.
Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.
Maintain technical documentation, operating procedures and knowledge-transfer materials.

Requirements & Qualifications

Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related field.
6–10 years of relevant data or software engineering experience, including at least 2 years working with cloud-based data platforms.
Strong hands-on experience in Python, PySpark, SQL, Apache Spark and Delta Lake.
Proven experience building and operating production data pipelines using Databricks and AWS.
Good knowledge of AWS services such as S3, Glue, Lake Formation, IAM, Lambda, Redshift and CloudWatch.
Experience with lakehouse architecture, data modelling, incremental processing, change data capture and data quality management.
Familiarity with workflow orchestration, CI/CD, Git and infrastructure-as-code tools such as Terraform.
Experience with pipeline monitoring, troubleshooting, performance optimisation and cloud security.
Ability to lead technical design, conduct code reviews, establish engineering standards and mentor engineers.
Strong problem-solving, communication and stakeholder management skills.

Preferred Skills

Experience with Unity Catalog, Databricks Workflows, Lakeflow, Delta Live Tables or Structured Streaming.
Familiarity with Airflow, dbt, Kafka, AWS DMS, MSK or Kinesis.
Experience building data products, data contracts, feature engineering or ML data pipelines.
Knowledge of Singapore public-sector data governance, security and compliance requirements.

Senior Data Engineer (AWS & Databricks) | Contract

Skills