Python/Spark/AI Developer
- Compensation
- Not disclosed
- Location
- Johannesburg, South Africa
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Full time
- Experience
- 2+ years
- Category
- Ai engineering
- Posted
Skills
Must have
- Bachelor’s degree in Computer Science, Engineering or a related discipline, or equivalent practical experience.
- Proven track record delivering production Python and Spark pipelines.
- Spark certification or equivalent credential is advantageous but not required.
Nice to have
- Experience engineering LLM integrations and AI-enabled data workflows.
- Experience building provider-neutral AI abstractions.
- Knowledge of prompt construction and structured output validation.
- Exposure to local or self-hosted inference using Ollama, vLLM, llama.cpp or LM Studio.
- Experience integrating hosted LLM APIs.
- Understanding of LLM evaluation and guardrails.
- Data governance and privacy engineering experience.
- Knowledge of PII tokenisation, hashing, k-anonymity and re-identification risk.
- Familiarity with POPIA and GDPR-related data handling.
- Multi-tenant data isolation awareness.
- Experience with lakehouse technologies including Hive Metastore, Trino and Apache Ranger.
- Understanding of catalogs, views and row/column-level data policies.
- Familiarity with the Azure data ecosystem, including ADLS Gen2, Synapse, ADF, Azure Key Vault and Service Bus.
- Observability experience using structured logging, OpenTelemetry or OpenLineage.
- Previous experience migrating legacy pipelines and performing detailed output parity comparisons.
- Databricks Certified Developer for Apache Spark or a comparable Spark certification.
Full description
Location: Hybrid / Remote
Employment Type: Full-Time
Industry: Data Engineering | AI | Software Development | Data Platforms
WatersEdge Solutions is partnering with a number of leading South African and international organisations to recruit an experienced Python/Spark/AI Developer to join a technically driven team working on the modernisation of large-scale data platforms.
This role will focus on replatforming legacy T-SQL workloads into modern Spark and Delta Lake pipelines, while building the APIs and AI capabilities that sit around them. You’ll work primarily in Python, with a strong emphasis on type-safe, tested and spec-driven development.
This is an excellent opportunity for an intermediate-to-senior developer who enjoys solving complex data engineering problems and wants to work across Python, distributed data processing, lakehouse architecture, APIs and emerging AI technologies.
About the Role
As Python/Spark/AI Developer, you’ll play a key role in migrating legacy data workloads onto a modern Spark-based architecture.
You’ll translate existing T-SQL logic into Spark SQL and PySpark pipelines, validate migrated workloads against legacy outputs, and ensure the new platform delivers reliable and demonstrable parity.
The engineering environment is heavily spec-driven. You’ll work autonomously from written requirements, design documentation and Architecture Decision Records (ADRs), with changes expected to include appropriate testing and documentation rather than code alone.
Alongside the core data engineering work, you’ll build REST APIs around pipeline jobs and contribute to AI capabilities, including provider-neutral LLM integrations and structured AI outputs.
Key Responsibilities
- Build production-grade Spark and PySpark pipelines on Delta Lake.
- Replatform legacy T-SQL workloads and stored-procedure-era logic into Spark SQL.
- Translate existing data logic accurately while maintaining expected business and data semantics.
- Write modern, type-hinted Python with strict static analysis.
- Develop and maintain automated pytest suites covering unit, integration and end-to-end testing.
- Validate migrated pipelines against legacy systems and provide clear parity and regression evidence.
- Build REST APIs using FastAPI or equivalent frameworks.
- Implement API request validation, authentication, idempotency and job-status semantics.