Data Platform Engineer
- Compensation
- Not disclosed
- Location
- Brisbane QLD
- Work setup
- Onsite
- Seniority
- Not specified
- Employment
- Contract
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Proven experience as a Senior Data Engineer, Data Platform Engineer, SDET, Data Quality Engineer, or similar technical role.
- Strong hands-on experience with Python, PySpark, and SQL in enterprise-scale environments.
- Demonstrated experience working with Databricks Lakehouse and large-scale distributed data processing platforms.
- Experience validating and testing complex data pipelines, data integrations, and distributed systems.
- Strong understanding of data quality, data validation, and automated testing methodologies.
- Experience with GitLab CI/CD and implementing automated quality gates within deployment pipelines.
- Experience working with Elasticsearch, event-driven architectures, and message brokers such as RabbitMQ.
- Knowledge of Infrastructure as Code (IaC) tools, preferably Terraform.
- Strong analytical, problem-solving, and troubleshooting skills.
- Excellent communication and stakeholder management skills with the ability to collaborate across multiple teams.
- Desirable Qualifications & Experience
- Experience with performance, load, scalability, resilience, and chaos testing.
- Experience working within cloud-native and modern data platform environments.
- Exposure to DevOps and Site Reliability Engineering (SRE) practices.
- Knowledge of C# and .NET technologies.
- Experience working in Agile delivery environments.
- Relevant certifications in Databricks, Cloud Technologies, or Data Engineering would be advantageous.
Full description
Company DescriptionMake extraordinary happen
At NCS Australia, we’re the challenger alternative in technology services and consulting. With more than 47 years of Australian heritage, we partner with governments and leading organisations to solve complex challenges, embrace new opportunities and deliver outcomes that make a real difference.
Our people bring together expertise across technology, consulting, delivery, operations, sales and corporate functions. We work collaboratively, challenge the status quo and give our people the opportunity to make an impact — for our clients, our communities and each other.
Job Description
We are seeking a Data Platform Engineer with strong quality engineering and automation expertise to support the development, validation, and reliability of large-scale data platforms. This role will focus on Databricks Lakehouse, PySpark, SQL, distributed data pipelines, event-driven architectures, and cloud-based data ecosystems, while also supporting automated testing, data validation, CI/CD quality gates, and platform resilience.
The successful candidate will play a key role in ensuring end-to-end data integrity across complex systems, validating data flows between applications and data platforms, and supporting performance, scalability, and reliability initiatives. Working closely with cross-functional teams, the individual will leverage strong Databricks, Python, and SQL expertise to investigate data quality issues, validate platform behaviour, and contribute to the continuous improvement of automation and engineering practices.
This is a hands-on technical role suited to a data engineering professional who can operate independently, collaborate across multiple teams, and apply quality engineering principles within a modern data platform environment. While testing and automation experience are important, the primary focus is deep expertise in Databricks, Python, SQL, and data platform engineering.
Responsibilities
- Design, develop, and maintain automated test frameworks and test suites.
- Build and enhance quality gates within GitLab CI/CD pipelines.
- Perform end-to-end testing of complex distributed data flows.
- Validate data integrity across RabbitMQ, Databricks Lakehouse, Lakebase, and Elasticsearch platforms.
- Conduct functional, integration, regression, and data validation testing.
- Collaborate with developers, platform engineers, and fellow SDETs to ensure seamless system integration.
- Coordinate cross-team testing activities across multiple application teams.
- Design and execute load, performance, and scalability testing strategies.