Umbrella Club
Senior Azure Data Engineer
Senior Data Engineer designing, building, and optimizing large-scale data platforms and complex ETL pipelines on the Azure stack — using Python, SQL, PySpark, Azure Synapse, Data Factory, and Azure Functions — with MLOps practices and DevOps tooling (Azure DevOps, Git) in an Agile, cross-functional team.
- Compensation
- Not disclosed
- Location
- Sydney, Australia
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 8+ years
- Category
- Data engineering
- Posted
Skills
Must have
- 8+ years of experience in software engineering/data engineering
- Strong hands-on experience with Python and SQL
- Extensive experience with Apache PySpark, Azure Synapse, Azure Data Factory (ADF), and Azure Functions
- Strong experience building complex ETL/data pipelines and modern data modelling concepts
- Experience with MS SQL, ADLS Gen2, NoSQL, and large-scale data warehousing
- Experience working with distributed systems and batch-processing environments
- Knowledge of MLOps architecture and practices
- Good understanding of Azure DevOps and Git
- Strong understanding of ETL and data warehousing methodologies
- Strong problem-solving and communication skills with experience working in Agile environments
Nice to have
- Java experience
Full description
We are looking for an experienced Senior Data Engineer to join our team and contribute to the design, development, and optimization of modern data platforms and large-scale data solutions.
Key Responsibilities
- Design, develop, and maintain complex data pipelines and ETL processes.
- Build scalable data solutions using Python, SQL, Apache PySpark, Azure Synapse, ADF, and Azure Functions.
- Develop and optimize data models, data warehouses, and large-scale data processing solutions.
- Work with SQL and NoSQL data stores, including MS SQL and ADLS Gen2.
- Develop and manage distributed data processing systems and clusters for batch processing.
- Implement data engineering solutions aligned with MLOps and modern data architecture practices.
- Support data integration, transformation, and migration initiatives.
- Follow Agile development practices and collaborate with cross-functional teams.
- Use modern DevOps tools such as Azure DevOps and Git to support development and deployment.
- Contribute to data platform performance, scalability, and reliability improvements.
About You
- 8+ years of experience in software engineering/data engineering.
- Strong hands‑on experience with Python and SQL; Java experience is desirable.
- Extensive experience with Apache PySpark, Azure Synapse, Azure Data Factory (ADF), and Azure Functions.
- Strong experience building complex ETL/data pipelines and working with modern data modelling concepts.
- Experience with MS SQL, ADLS Gen2, NoSQL, and large‑scale data warehousing.
- Experience working with distributed systems and batch‑processing environments.
- Knowledge of MLOps architecture and practices.
- Good understanding of Azure DevOps and Git.
- Strong understanding of ETL and data warehousing methodologies.
- Strong problem‑solving and communication skills with experience working in Agile environments.
Location: Sydney