Senior Databricks AI/ML Engineer

We are looking for an experienced Senior Databricks AI/ML Engineer with expertise in Databricks, Apache Spark, Machine Learning, and Generative AI. The ideal candidate should have hands-on experience building scalable AI/ML solutions, modern data platforms, and advanced analytics pipelines on the Databricks Lakehouse platform. Experience with Teradata, MLflow, and LLM integration is highly desirable.

Key Responsibilities:

• Design, develop, and optimize AI/ML solutions using the Databricks Lakehouse Platform.

• Build scalable data pipelines using Apache Spark (PySpark/Spark SQL) and Delta Lake.

• Develop and deploy machine learning models using MLflow and Databricks Machine Learning.

• Build and integrate LLM-based and Generative AI solutions for enterprise use cases.

• Perform feature engineering, model training, evaluation, and deployment for predictive analytics and time-series use cases.

• Design and optimize SQL-based analytics and reporting solutions.

• Support Teradata integration, migration, and performance optimization where required.

• Monitor and optimize cluster performance, workloads, and resource utilization on Databricks.

• Develop reusable Python libraries and automation scripts for data engineering and ML workflows.

• Collaborate with Data Scientists, Data Engineers, Product teams, and business stakeholders to deliver scalable AI-driven solutions.

• Implement CI/CD, model governance, and MLOps best practices.

• Prepare technical documentation and ensure compliance with security and data governance standards.

Required Skills:

• 8-13 years of experience in Data Engineering, Machine Learning, or AI Engineering.

• Strong hands-on expertise with the Databricks Platform (Workspaces, Clusters, Jobs, Unity Catalog, Databricks SQL).

• Extensive experience with Apache Spark (PySpark/Spark SQL) and Delta Lake.

• Strong programming skills in Python, including Pandas, NumPy, and data visualization libraries.

• Advanced proficiency in SQL and large-scale data processing.

• Experience with MLflow, model lifecycle management, and MLOps practices.

• Experience integrating Large Language Models (LLMs) and Generative AI into enterprise applications.

• Strong knowledge of feature engineering, predictive modeling, and time-series analytics.

• Experience with Teradata Administration, DBQL, System Views, and performance tuning.