nexbridge recruitment pte. ltd.
Senior Data Engineer (AWS & Databricks) | Contract
Contract role for a Senior Data Engineer to design, build and maintain batch and streaming data pipelines on AWS and Databricks, including lakehouse layers with Spark/PySpark, Delta Lake, data quality, CI/CD and governance for a Singapore public-sector agency, while mentoring junior engineers.
- Compensation
- Not disclosed
- Location
- Singapore, Singapore
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Contract
- Experience
- 2+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or related field
- 6-10 years of relevant data or software engineering experience, including 2+ years with cloud-based data platforms
- Strong hands-on experience in Python, PySpark, SQL, Apache Spark and Delta Lake
- Proven experience building and operating production data pipelines using Databricks and AWS
- Good knowledge of AWS services such as S3, Glue, Lake Formation, IAM, Lambda, Redshift and CloudWatch
- Experience with lakehouse architecture, data modelling, incremental processing, change data capture and data quality management
- Familiarity with workflow orchestration, CI/CD, Git and infrastructure-as-code tools such as Terraform
- Experience with pipeline monitoring, troubleshooting, performance optimisation and cloud security
- Ability to lead technical design, conduct code reviews, establish engineering standards and mentor engineers
- Strong problem-solving, communication and stakeholder management skills
Nice to have
- Experience with Unity Catalog, Databricks Workflows, Lakeflow, Delta Live Tables or Structured Streaming
- Familiarity with Airflow, dbt, Kafka, AWS DMS, MSK or Kinesis
- Experience building data products, data contracts, feature engineering or ML data pipelines
- Knowledge of Singapore public-sector data governance, security and compliance requirements
- Relevant Databricks or AWS certifications
Full description
Key Responsibilities
- Design, develop and maintain scalable batch and streaming data pipelines across AWS and Databricks.
- Build and manage lakehouse architectures using Apache Spark, PySpark and Delta Lake, including Bronze, Silver and Gold data layers.
- Integrate data from databases, APIs, files and event streams using incremental ingestion and change data capture.
- Implement data quality checks, schema validation, reconciliation, metadata management and data lineage.
- Develop automated workflows and CI/CD pipelines using Databricks Workflows, AWS services, Git and infrastructure-as-code tools.
- Monitor pipeline performance, troubleshoot issues and implement logging, alerts and recovery procedures.
- Optimise Spark and SQL workloads for performance, reliability and cloud cost efficiency.
- Implement data security, access controls, encryption and governance in line with agency requirements.
- Establish engineering standards, review code and architecture, and mentor junior engineers.
- Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.
- Maintain technical documentation, operating procedures and knowledge-transfer materials.
Requirements & Qualifications
- Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related field.
- 6-10 years of relevant data or software engineering experience, including at least 2 years working with cloud-based data platforms.
- Strong hands-on experience in Python, PySpark, SQL, Apache Spark and Delta Lake.
- Proven experience building and operating production data pipelines using Databricks and AWS.
- Good knowledge of AWS services such as S3, Glue, Lake Formation, IAM, Lambda, Redshift and CloudWatch.
- Experience with lakehouse architecture, data modelling, incremental processing, change data capture and data quality management.
- Familiarity with workflow orchestration, CI/CD, Git and infrastructure-as-code tools such as Terraform.
- Experience with pipeline monitoring, troubleshooting, performance optimisation and cloud security.
- Ability to lead technical design, conduct code reviews, establish engineering standards and mentor engineers.
- Strong problem-solving, communication and stakeholder management skills.
Preferred Skills
- Experience with Unity Catalog, Databricks Workflows, Lakeflow, Delta Live Tables or Structured Streaming.