Lead Data Engineer (Spring, TX)
Lead Data Engineer who hands-on designs and builds scalable Lakehouse data products on Databricks and Microsoft Azure, using Spark, Delta Lake, Azure Data Factory and dbt. Also mentors engineers, leads technical architecture, and drives adoption of Databricks AI/agentic capabilities like Genie and Agent Bricks.
- Compensation
- Not disclosed
- Location
- Not listed
- Work setup
- Not specified
- Seniority
- Lead
- Employment
- Not specified
- Experience
- 7+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Bachelor's degree in Computer Science, Information Systems, Engineering, Data Analytics, or related field, or equivalent professional experience
- 7+ years of experience in Data Engineering, Data Platform Engineering and Data Architecture
- Demonstrated experience leading technical architecture and solution design initiatives
- Expert-level knowledge of Databricks, Delta Lake, Apache Spark, Spark SQL, SQL, Python/PySpark, and Azure Data Factory
- Strong expertise in dimensional modeling, data warehouse design, and semantic modeling
- Experience implementing enterprise Lakehouse and data platform architectures
- Hands-on experience with Unity Catalog, Delta Live Tables (DLT), Auto Loader, Databricks Workflows, DBT, and Lakeflow Pipelines
- Experience implementing CI/CD pipelines and DevOps practices using Azure DevOps, Git, and Infrastructure as Code
- Strong understanding of data quality, governance, observability, monitoring, and operational support
- Excellent analytical, problem-solving, communication, and stakeholder management skills
- Proven experience mentoring engineers and leading technical delivery teams
Nice to have
- 7-10+ years of related experience
- Experience implementing AI-driven and agentic solutions within Databricks
- Experience with Databricks Mosaic AI, Vector Search, Agent Bricks, Genie, and Databricks Apps
- Experience building Retrieval-Augmented Generation (RAG) and semantic retrieval solutions
- Knowledge of Azure Synapse Analytics and modern cloud-native data architectures
- Experience supporting Power BI semantic models and enterprise reporting environments
- Azure and Databricks certifications
- Experience designing reusable engineering frameworks and platform accelerators
- Experience developing AI-assisted development tooling, reusable skills, or code generation capabilities
Full description
Job Summary:
We are seeking an experienced and highly motivated Lead Data Engineer to design, develop, and implement scalable data products and solutions utilizing the Databricks Data Intelligence Platform and Microsoft Azure. This role is ideal for a hands-on technical individual who combines deep expertise in data engineering, dimensional modeling, platform development, and modern data engineering practices and agentic code generation with a passion for emerging capabilities within the Databricks ecosystem.
The successful candidate will be responsible for developing reusable engineering frameworks, implementing enterprise data solutions, mentoring engineering teams, and driving adoption of modern Databricks technologies including Azure Data Factory, Lakeflow, Lakebase, Agentbricks, Genie, Genie Spaces, Genie Code, Databricks Apps, and AI Agent Frameworks. Working within established enterprise architecture standards, this individual will play a critical role in delivering scalable Lakehouse solutions that support analytics, reporting, artificial intelligence, machine learning, and agentic applications across the organization. The Lead Data Engineer will collaborate closely with Product Managers, Project Managers, Data Engineers, Analytics teams, and business stakeholders to build modern, scalable, secure, and high-quality data products that drive business value.
Required Qualifications
-
Bachelor’s degree in computer science, Information Systems, Engineering, Data Analytics, or a related field, or equivalent professional experience.
- 7+ years of experience in Data Engineering, Data Platform Engineering and Data Architecture.
- Demonstrated experience leading technical architecture and solution design initiatives.
- Expert-level knowledge of:
- Databricks
- Delta Lake
- Apache Spark
- Spark SQL
- SQL
- Python / PySpark
- Azure Data Factory
- Strong expertise in dimensional modeling, data warehouse design, and semantic modeling.
- Experience implementing enterprise Lakehouse and data platform architectures.
- Hands-on experience with:
- Unity Catalog
- Delta Live Tables (DLT)
- Auto Loader
- Databricks Workflows
- DBT
- Lakeflow Pipelines
- Experience implementing CI/CD pipelines and DevOps practices using Azure DevOps, Git, and Infrastructure as Code.
- Strong understanding of data quality, governance, observability, monitoring, and operational support.