Data Engineer
Senior data engineer (6-10 years) building and operating production-scale batch/streaming data pipelines on AWS and Databricks. Core stack: Python, PySpark, SQL, Apache Spark, Delta Lake, plus orchestration, CI/CD, Terraform, data quality and pipeline observability, with mentoring and technical leadership.
- Compensation
- Not disclosed
- Location
- Singapore
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- 2+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or related technical discipline
- 6-10 years of relevant data or software engineering experience
- Hands-on design and operation of production data pipelines
- At least 2 years working with cloud-based data platforms
- Technical leadership through architecture, engineering standards, code review, mentoring or ownership of critical data services
- Strong Python, PySpark and SQL including distributed processing, complex transformations, query optimisation, debugging and automated testing
- Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming data engineering
- Hands-on AWS data engineering experience (S3, Glue, Lake Formation, IAM, KMS, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis)
- Experience designing lakehouse or data-lake architectures including layered data patterns and dimensional/domain data modelling
- Experience with incremental ingestion, change data capture, idempotency, late-arriving data handling, backfills, replay and recovery
- Experience implementing data quality, schema validation, reconciliation, metadata and lineage as automated parts of pipelines
- Experience with workflow orchestration, Git-based development, CI/CD, infrastructure as code (e.g. Terraform) and environment promotion controls
- Ability to implement pipeline observability, dashboards, alerting, runbooks, incident diagnosis and service recovery
Nice to have
- Experience with Databricks Unity Catalog, Lakeflow/Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing, Databricks Asset Bundles
- Experience with Apache Airflow, dbt, Kafka, Amazon MSK/Kinesis or AWS DMS for ingestion and orchestration
- Experience designing data products, data contracts, semantic layers, master/reference data or data mesh-aligned models
- Experience supporting feature engineering, ML training datasets, feature stores or MLOps pipelines on Databricks and AWS
- Experience in Singapore public-sector environments (IM8-aligned controls, data classification, VAPT, system assurance, cross-agency data sharing)
- Certifications such as Databricks Certified Data Engineer Professional, AWS Certified Data Engineer - Associate, AWS Certified Solutions Architect
Full description
Avensys is a reputed global IT professional services company headquartered in Singapore. Our service spectrum includes enterprise solution consulting, business intelligence, business process automation and managed services. Given our decade of success, we have evolved to become one of the top trusted providers in Singapore and service a client base across banking and financial services, insurance, information technology, healthcare, retail and supply chain.
We are currently looking to hire highly skilled Data Engineer. This is an exciting opportunity to expand your skill set, achieve job satisfaction and work-life balance. More details as below.
Requirements & Qualifications
Educational Qualification & Experience
Education: Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related
technical discipline.
Experience: Typically 6–10 years of relevant data or software engineering experience, including hands-on design and
operation of production data pipelines and at least 2 years working with cloud-based data platforms. Candidates
should demonstrate technical leadership through architecture, engineering standards, code review, mentoring or
ownership of critical data services.
B. Technical & Professional Competencies
Essential (Must-Have)
Strong Python, PySpark and SQL capability, including distributed processing, complex transformations, query
optimisation, debugging and automated testing.
Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming
data engineering.
Hands-on AWS data engineering experience with a relevant mix of S3, Glue, Lake Formation, IAM, KMS, Lambda,
Step Functions, EventBridge, CloudWatch, Redshift and MSK/Kinesis.
Experience designing lakehouse or data-lake architectures, including layered data patterns, dimensional or
domain data modelling and reusable curated datasets.
Experience implementing incremental ingestion, change data capture, idempotency, late-arriving data handling,
backfills, replay and recovery.
Experience implementing data quality, schema validation, reconciliation, metadata and lineage as automated
parts of production pipelines.
Experience with workflow orchestration, Git-based development, CI/CD, infrastructure as code such as
Terraform, and environment promotion controls.
Ability to implement pipeline observability, operational dashboards, alerting, runbooks, incident diagnosis and