Databricks Architect (Lead Data Platform Architect)

Role Overview

We are looking for a highly skilled Databricks Architect to design, build, and scale enterprise-grade Lakehouse data platforms. This role will drive architecture strategy, platform standardization, and enterprise data modernization initiatives, leveraging Databricks and cloud ecosystems.

The ideal candidate brings deep expertise in Spark, Delta Lake, and cloud-native architecture, along with strong leadership in driving large-scale data transformations.

Key Responsibilities

Data Platform Architecture

• Define and implement end-to-end Databricks Lakehouse architecture.

• Design scalable systems for:
• Batch & real-time data processing

• Structured & unstructured workloads

• Establish medallion architecture (Bronze, Silver, Gold layers) as a standard.

Databricks Platform Leadership

• Lead deployment and optimization of:
• Azure Databricks / AWS Databricks / GCP Databricks

• Define standards for:
• Workspace design & cluster strategy

• Job orchestration

• Data storage (Delta Lake)

• Drive adoption of:
• Unity Catalog

• MLflow

• Databricks SQL & Photon

Solution Design & Engineering

• Architect robust data ingestion frameworks:
• Batch (ADF, Airflow)

• Streaming (Kafka, Event Hub)

• Define reusable patterns for:
• ETL/ELT pipelines

• Data modeling (star schema, data vault, dimensional models)

• Guide engineering teams on best practices in Spark/PySpark optimization.

Performance & Cost Optimization

• Optimize workloads for:
• Query performance

• Cluster utilization

• Storage efficiency

• Implement cost governance strategies (auto-scaling, job clusters, spot instances).

Data Governance & Security

• Architect enterprise-grade governance frameworks:
• Data lineage, cataloging, metadata management

• Fine-grained access control (RBAC/ABAC)

• Ensure compliance with data privacy and regulatory standards.

Cloud & Ecosystem Integration