Data Engineering Architect - Databricks

Primary Responsibilities :

• Define and drive the Databricks Data Engineering architecture and strategy, aligned with enterprise Data & Analytics standards.

• Take end-to-end ownership of Databricks solutions from architecture and design through implementation, production deployment, support, and optimization.

• Define and enforce policy-based governance and cost controls across Databricks workloads, including Apps, Agents, compute, AI/ML resources, and AI Gateway.

• Establish and enforce RBAC and least-privilege security across Unity Catalog, Databricks Apps, Agents, AI Gateway, service principals, and Dev/QA/Production environments.

• Design reusable data ingestion and transformation frameworks using Databricks, Spark, Delta Lake, SQL, Python/PySpark, and Lake flow.

• Architect scalable Lakehouse solutions on Databricks and AWS supporting batch, streaming, near-real-time, analytics, and AI use cases.

• Design and support Agentic AI and Databricks Apps development, including architecture, deployment, integration, security, monitoring, and production readiness.

• Establish and enforce Data Engineering Governance and Security using Unity Catalog, including RBAC, least privilege, service principals, data access controls, lineage, auditing, and data quality.

• Define engineering and governance standards across Dev, QA, and Production, including Git, Branches etc.

• Design data integration patterns for SAP/ERP, databases, APIs, files, streaming platforms, and cloud services.

• Design and evaluate data models and serving patterns for BI, analytics, Data Science, AI, applications, and downstream consumers.

• Establish standards for Databricks compute, performance optimization, monitoring, reliability, and cost management.

• Proactively identify and resolve architecture, security, governance, performance, and production issues.

• Work closely with Security, Cloud, Infrastructure, Data Governance, Data Science, BI, application, and business teams to deliver enterprise solutions.

• Provide technical leadership, architecture reviews, hands-on prototyping, engineering standards, and mentoring.

• Demonstrate a strong ownership mindset by independently driving initiatives, managing dependencies and risks, and ensuring successful delivery.

You Must Have :

• 9+ years of Data Engineering/Data Architecture experience with 4+ years of hands-on Databricks experience.

• Strong experience with Databricks, Delta Lake, Spark, SQL, Python/PySpark, and Lakehouse architecture.