Central Business Solutions, Inc
Senior Databricks & AI Architect/Lead - Only W2 - (JK)
Long-term W2 contract role (via Central Business Solutions) for a Senior Databricks & AI Architect to design and govern a modern data platform on the Databricks Data Intelligence Platform, build scalable Spark-based architectures, and lead Agentic AI/LLM work (LangChain, LlamaIndex, RAG) for insurance policy and regulatory data. Onsite initially, converting to hybrid.
- Compensation
- Not disclosed
- Location
- Dallas, TX/NJ/NY/ California, CA, TX, NJ, NY
- Work setup
- Not specified
- Seniority
- Lead
- Employment
- Contract
- Experience
- 3+ years
- Category
- Architecture
- Posted
Skills
Must have
- 8+ years in Data Engineering/Architecture
- 3+ years hands-on leadership within the Databricks ecosystem
- Expert-level knowledge of the Spark framework (PySpark/Scala)
- Delta Lake
- Databricks SQL
- Proven experience building and deploying AI solutions
- Deep understanding of the Agentic AI landscape (autonomous, goal-oriented agents)
- Strong background in data modelling and creation of reusable ETL/ELT templates and frameworks
- Design end-to-end data architectures on the Databricks Data Intelligence Platform
- LLM orchestration with LangChain/LlamaIndex and RAG patterns for insurance policy and regulatory documents
Nice to have
- Medallion architecture data engineering templates
- Data modelling with Star/Snowflake and Vault 2.0 for structured and unstructured data
- Governance and compliance with Databricks Unity Catalog
Full description
**Title: Senior Databricks & AI Architect/Lead
Location: Dallas, TX/NJ/NY/ California (Initial onsite and can converted to Hybrid then after)
Duration: Long Term contract role**
We are seeking a visionary Senior Databricks & AI Architect to lead the evolution of our modern data platform and AI roadmap.
You will be responsible for designing scalable Spark-based architectures.
Key Responsibilities
- Architectural Leadership: Design and govern end-to-end data architectures on the Databricks Data Intelligence Platform.
- AI Innovation: Lead the design of emerging AI workflows, specifically focusing on Agentic AI (multi-agent systems), LLM orchestration (LangChain/LlamaIndex), and RAG patterns tailored for insurance policy and regulatory documents.
- Data Engineering Excellence: Standardise data engineering templates (Medallion architecture) and implement robust data modelling (Star/Snowflake, Vault 2.0) to handle structured and unstructured insurance data.
- Governance & Security: Ensure all designs comply with industry regulations using Databricks Unity Catalog.
Required Skills & Experience
- Experience: 8+ years in Data Engineering/Architecture, with at least 3+ years of hands-on leadership within the Databricks ecosystem.
- Core Tech: Expert-level knowledge of the Spark framework (PySpark/Scala), Delta Lake, and Databricks SQL.
- AI/ML Expertise: Proven experience building and deploying AI solutions. Deep understanding of the Agentic AI landscape—moving beyond simple chatbots to autonomous, goal-oriented agents.
- Data Management: Strong background in data modelling and the creation of reusable ETL/ELT templates and frameworks.