Principal Architect - Data Platform

Principal Architect : Experience : 1215 years.

Location : Pune (onsite).

About the Role :

We are looking for a Principal Architect to lead the design and development of a Lakehouse solution end to end.

This is an opportunity to architect and build a reusable data platform framework from the ground up.

You will own the technical architecture, write code for the core framework, mentor a growing engineering team, and work closely with leadership on the overall technology direction.

The ideal candidate is a hands-on architect with a proven track record of building complex frameworks and platforms using open-source big-data technologies such as Spark, Hadoop, and modern Lakehouse stacks (e.g., Iceberg, Trino, Databricks, etc).

Key Responsibilities : 1. Architecture (40%) :

• Own the end-to-end architecture of the Lakehouse platform and its reusable framework components.

• Design a metadata-driven data engineering framework built on Spark SQL and PySpark.

• Architect a pipeline engine supporting dynamic SQL generation, incremental loads, CDC, SCD implementations, and schema evolution.

• Design Kubernetes-native deployment with multi-tenancy, resource isolation, scaling strategy, high availability, and disaster recovery.

• Define API contracts, plugin/extension architecture, and integration patterns.

• Drive and document key architecture decisions (ADRs).

2. Hands-on Engineering (40%) :

• Build core framework components : pipeline engine, dynamic SQL generation, data quality checks, error handling and retry mechanisms.

• Implement observability across the platform.

• Optimize Spark job performance (partitioning, shuffle tuning, memory management, query plans).

• Establish and implement CI/CD pipelines, automated testing (unit and integration), and release practices.

• Embed data governance and security into the platform (access control, auditability, cataloging).

3. Engineering Leadership (20%) :

• Mentor and guide junior engineers; conduct design and code reviews.

• Define coding standards, engineering best practices, and documentation norms.

• Contribute to broader technical direction and team growth.

• Continuously raise the bar on engineering quality and delivery practices.

Required Experience :

• 12 to 15 years of software engineering experience overall.