Design and govern enterprise AI architectures on Databricks, including MLOps pipelines, Unity Catalog governance, and generative AI use cases like RAG and LLM integration.
Compensation
Not disclosed
Location
Johannesburg, South Africa
Work setup
Not specified
Seniority
Not specified
Employment
Not specified
Experience
5+ years
Category
Architecture
Posted
Skills
agile
ai
analytics
archimate
aws
azure
big-data
ci-cd
cloud
cloud-native
data-engineering
data-lake
data-mesh
data-modeling
data-pipelines
data-warehousing
databricks
delta-lake
devops
elt
etl
gcp
generative-ai
hadoop
lakehouse
llm
machine-learning
master-data-management
mdm
metadata-management
mlflow
mlops
model-deployment
python
requirements-gathering
solution-design
spark
sql
test-automation
togaf
unity
unity-catalog
vector-databases
Must have
Matric
Degree or diploma in Information Technology, Computer Science, Engineering OR relevant diploma / degree
Experience: Requires a minimum of 5 years in a technical/solution design role and a minimum of 7 years relevant IT experience
Data and AI Experience: Required a minimum of 7 years related experience in AI, data engineering, data modeling and design and data management and governance
Expert-level proficiency in Databricks, including Delta Lake, Spark, and MLflow.
Full description
The key focus for the senior data/AI architect is to perform planning aligned to key AI solutions, build and participate in the architecture capability building, perform AI architecture and design, manage AI architecture risk and compliance, provide design and build governance and support and communicate and share knowledge around the architecture practices, guardrails, blueprints and standards related to the AI solution design.
A key focus of this role is partnering with the AI Technology Centre of Excellence to build out the organisation's Databricks AI platform and support the delivery of enterprise AI and generative AI use cases.
Planning
Lead AI solution
requirements gathering and ensure alignment with business objectives and constraints.
Define and refine AI
architecture runways for intentional architecture with the key stakeholders
Provide input into
business cases and costing
Participate and
provide AI architectural runway requirements into Programme Increment (PI) Planning
Architecture Capability
Design and implement enterprise-grade AI
architectures leveraging Databricks and cloud-native technologies.
Develop and
oversee AI architecture views and ensure alignment with enterprise architecture.
Maintain and oversee
the AI solution artifacts in the set enterprise repository and knowledge portals aligned to the rest of the architecture
Manage the AI
architecture processes based on the requirements for each architype
Manage change impact
of the AI architecture with stakeholders
Develop and
participate in the build of the AI architecture practice with embedded architects and engineers including the relevant methods, repository and tools
Manage the AI
architecture considering the business, application, information/data and technology viewpoints
Establish, enforce and
implement AI standards, guardrails, frameworks, and patterns
Data/AI Architect (Databricks) at Blue Pearl (Johannesburg, South Africa) — aijobsearch
Partner with the AI
Tech COE to define and evolve the Databricks AI platform architecture, ensuring alignment with enterprise data and AI strategy
Design and implement
AI/ML architectures on Databricks, including MLOps pipelines, model lifecycle management, Unity AI Gateway and Unity Catalog governance for AI/ML assets
Solution Design
Lead and
review logical and detailed AI architecture
Evaluate and
approve AI solution options and technology selections
Select appropriate
technology, tools and build for the solution
Oversee and
maintain the AI solution blueprints
Drive incremental
modernisation initiatives in the delivery area
Design and evaluate
architectures for AI and generative AI use cases, including RAG pipelines, vector stores, feature stores, and LLM integration patterns
Risk, Governance and Compliance
· Identify, assess and mitigate risks at a AI solution architecture level
· Ensure and enforce compliance with policies, standards, and regulations
· Lead AI architecture reviews and integrate with governance functions
· Integrate with other governance and compliance functions to ensure continuity in managing the investment and risk for the organisation pertaining to the solution architectures
· Establish and provide AI standards, guidance, and tools to delivery teams.
Implementation and Collaboration
· Establish and provide AI solution architectures and tools to the delivery and AI engineering teams
· Lead and facilitate collaboration with delivery teams to achieve architecture objectives
· Manage and resolve deviations and ensure up-to-date AI solution design documentation
· Identify opportunities to optimise delivery of solutions
· Oversee and conduct post-implementation reviews
· Ensure the AI architecture supports CI/CD pipelines to facilitate rapid and reliable deployment of data solutions
· Implement automated testing frameworks for AI solutions to ensure quality and reliability throughout the development lifecycle.
· Establish performance monitoring and optimisation practices to ensure AI solutions meet performance benchmarks and can scale as needed.
· Integrate robust AI security measures, including encryption, access controls, and regular security audits, into the implementation process.
Communication and Knowledge Sharing
· Communicate and advocate up-to-date AI solution architecture views
· Communicate the relevant AI standards, practices, guardrails and tools to stakeholders relevant to the solution design
· Ensure IT teams are well-informed and trained in architecture requirements
· Communicate and collaborate with stakeholders' relevant views on planning, technology assessments, risk, compliance, governance and implementation assessments
· Foster collaboration between AI architects, AI engineers, and other IT teams through regular cross-functional meetings and agile ceremonies.
· Communicate and maintain up-to-date blueprint designs for key data solutions
· Ensure effective participation in the agile ceremonies (PI planning, sprint planning, retrospectives, demos)
· Implement regular feedback loops with stakeholders and end-users to continuously improve data solutions based on real-world usage and requirements
· Create a culture of knowledge sharing by organising regular workshops, training sessions, and documentation updates to keep all team members informed about the latest AI architecture practices and tools
Requirements
**MINIMUM QUALIFICATIONS/EXPERIENCE **
Matric
Degree or diploma in
Information Technology, Computer Science, Engineering OR relevant diploma / degree
Experience: Requires a minimum of
5 years in a technical/solution design role and a minimum of 7 years relevant IT experience
Data and AI
Experience:Required a minimum of 7 years related experience in AI, data engineering, data modeling and design and data management and governance
Expert-level proficiency in Databricks, including
Delta Lake, Spark, and MLflow.
· Proven experience architecting and delivering AI/ML solutions on Databricks, including MLOps, model deployment and monitoring, and Unity Catalog governance for AI/ML assets.
· Hands-on experience in large-scale data and AI platform implementation (preferably cloud-based).
**ADDITIONAL QUALIFICATIONS/EXPERIENCE (PREFERRED, NOT A REQUIREMENT)**
DAMA-DMBOK
TOGAF
ArchiMate
Cloud Certifications
(AWS, Azure)
Financial Industry
Experience
Certifications in Databricks, AWS ML, AWS Data
Engineering or similar.
Experience with
generative AI / LLM architectures (e.g. RAG pipelines, vector databases, AI gateways)
Databricks Certified
Machine Learning Professional or equivalent certification
**Data Related Experience: **
· Big Data and Analytics (e.g., Hadoop, Spark)
· Data Warehousing
· Master Data Management (MDM)
· Data Lakes, Lakehouse, and Data Mesh
· Metadata Management
· ETL/ELT Processes
· Data Privacy and Compliance
Cloud Data Services
Experience with AI cloud platforms (Azure, AWS,
or GCP) and associated data services.
· Proficiency in SQL, Python, and distributed data processing frameworks.
· Familiarity with CI/CD for data pipelines and DevOps practices.
· Experience with Lakehouse architecture and real-time streaming solutions
**Related attributes and competencies related to architecture: **
· Critical thinking/problem solving
· Teamwork/collaboration
· Effective Communication Skills
· Leadership skills
· Knowledge and experience in architecture domains
· Knowledge and experience in architecture methods, frameworks and tools
· Solution Design Experience
· Agile Knowledge and Experience
· Cloud Knowledge and Experience
**AI related competencies: **
· AI architecture principles and methodologies
· AI integration technologies and tools
· AI management and governance
· AI/ML architecture, MLOps, and model lifecycle management knowledge and experience