Senior Data Engineer
Senior Data Engineer placed by DFP Recruitment (apparently for a government agency, as Australian citizenship is required) to design, build and maintain AWS-based batch and event-driven data pipelines and data products supporting data science, ML and analytics. Core stack: Python, advanced SQL, S3/Glue/Lambda/Athena, Terraform/CloudFormation, Docker/ECS/EKS. Roles based in NSW/VIC.
- Compensation
- Not disclosed
- Location
- Melbourne, Australia
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Australian citizenship
- Experience delivering production-quality data pipelines and data products in AWS (S3, Glue, Lambda, Athena, EMR, Kinesis, Step Functions)
- Strong experience with ETL/ELT patterns, data modelling, orchestration, data quality testing, metadata and lineage
- Modern software engineering and DevOps practices: Git, code review, automated testing, CI/CD, IaC with Terraform or CloudFormation
- Strong Python programming with maintainable, modular, testable production code (Java or Scala relevant)
- Advanced SQL including query optimisation and performance tuning on large datasets
- Hands-on experience with containerised environments (Docker, ECS, EKS) plus security, access controls, secrets management and monitoring
Nice to have
- Experience supporting data science, ML, generative AI or document intelligence solutions (feature pipelines, vector/search retrieval, evaluation datasets)
- Experience with modern data platforms such as Snowflake, Databricks, Redshift, Spark or dbt
- Knowledge of MLOps/DataOps practices including versioning, reproducibility and automated deployment
- SAS Base Programming
Full description
- Position available in the following states: NSW, VIC
- Australian Citizenship Required
DFP Recruitment is seeking an a Senior Data Engineer who will design, build and maintain reliable data pipelines and data products that support data science, machine learning, artificial intelligence and advanced analytics. Working closely with data scientists, machine learning engineers, solution architects, platform teams and business stakeholders, you will translate complex data requirements into well-engineered datasets and services, taking solutions from exploration and prototyping through to secure, repeatable and supportable production delivery. You will apply strong software engineering, cloud and DevOps practices to ensure data quality, lineage, security, privacy and monitoring, supporting the responsible use of data.
Key duties and responsibilities
- Design, build, test and maintain scalable batch and event-driven data pipelines for analytics, machine learning and AI use cases.
- Develop curated, reusable and well-documented datasets and data products from structured and unstructured sources.
- Work with data scientists and machine learning engineers to productionise data preparation, feature engineering, model input and model output pipelines.
- Translate business and analytical requirements into technical designs, delivery estimates, work items and acceptance criteria.
- Implement automated data quality controls, validation, metadata, lineage, logging, monitoring and alerting.
- Optimise SQL, Python and cloud workloads for performance, reliability, security and cost efficiency.
- Apply infrastructure-as-code, version control, automated testing, CI/CD and environment management practices to data engineering solutions.
- Prepare and maintain technical documentation, including data mappings, interface specifications, runbooks, support procedures and architectural decisions.
- Collaborate with Agency data custodians, cyber security, privacy, platform, architecture and delivery teams to ensure solutions meet governance and operational requirements.
Technical skills
- AWS data engineering (S3, Glue, Lambda, Athena, EMR, Kinesis and Step Functions)
- Python and advanced SQL; ETL/ELT, data modelling, orchestration, quality testing, metadata and lineage; Git, automated testing, CI/CD and infrastructure-as-code using Terraform or AWS CloudFormation; Docker, ECS or EKS; security, access controls, secrets management and monitoring.
Essential criteria
- Demonstrated experience designing and delivering production-quality data pipelines and data products in AWS, using services such as S3, Glue, Lambda, Athena, EMR, Kinesis, Step Functions or equivalent managed services.
- Strong experience with ETL/ELT patterns, data modelling, orchestration, data quality testing, metadata and lineage across structured and unstructured data.