Watershed Consulting
Data Platform Engineer - - Azure & Databricks
12-month renewable contract for a Data Platform Engineer in Sandton, Johannesburg (3 days per week onsite): designing, building, and operating an Azure + Databricks data platform using Terraform infrastructure-as-code, CI/CD pipelines (GitHub Actions/Azure DevOps), and Python/.NET automation.
- Compensation
- Not disclosed
- Location
- Johannesburg, South Africa
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Solid hands-on experience with Microsoft Azure infrastructure components (Entra ID, Key Vault, networking, storage)
- Practical experience with Azure Databricks, including Unity Catalog and cluster/job management
- Proven experience authoring Terraform in multi-environment setups for reproducible deployments
- Proficiency in Python
- Working knowledge of SQL
- Experience building and maintaining CI/CD pipelines using GitHub Actions, Azure DevOps, or similar tools
- Good understanding of data platform governance, cost optimization, security controls, and monitoring practices
- Strong troubleshooting skills across networking, authentication, and pipeline integrations
- Excellent collaboration and documentation skills for cross-functional platform delivery
Nice to have
- .NET experience
Full description
Job Title
Data Platform Engineer — Sandton, Johannesburg (3 days per week onsite)
Duration: 12 months (renewal)
Description
Purpose: Support the design, build, and operation of the Data Platform, blending infrastructure engineering with data engineering practices to keep the Databricks and Azure environment reliable, automated, and self-service for platform users.
Responsibilities
- Design, implement and maintain CI/CD pipelines (GitHub Actions, Azure DevOps or similar) for infrastructure and application deployments.
- Write and maintain Terraform infrastructure-as-code for Azure services and Azure Databricks, including Unity Catalog, workspaces, networking, and compute policies.
- Develop and support platform tooling and automation using Python and/or .NET.
- Build and maintain shared libraries or modules consumed by data engineers and analysts across the platform.
- Design and implement management pipelines and processes for platform concerns such as cost tracking, governance, provisioning, and monitoring.
- Collaborate with data engineers, analysts, security, and infrastructure teams to translate requirements into platform capabilities and secure designs.
- Troubleshoot infrastructure, pipeline, and integration issues, including networking, authentication, and connectivity problems.
- Contribute to platform documentation, runbooks, and best-practice standards to enable self-service for platform users.
Qualifications
- Solid hands-on experience with Microsoft Azure infrastructure components (Entra ID, Key Vault, networking, storage).
- Practical experience with Azure Databricks, including Unity Catalog and cluster/job management.
- Proven experience authoring Terraform in multi-environment setups for reproducible deployments.
- Proficiency in Python; working knowledge of SQL. .NET experience is a strong advantage.
- Experience building and maintaining CI/CD pipelines using GitHub Actions, Azure DevOps, or similar tools.
- Good understanding of data platform governance, cost optimization, security controls, and monitoring practices.
- Strong troubleshooting skills across networking, authentication, and pipeline integrations.
- Excellent collaboration and documentation skills for cross-functional platform delivery.
Skills
- Microsoft Azure
- Entra ID
- Azure Key Vault