Sr Data Engineer- Databricks

Exp- 5+ yrs

Location- Remote (Preferred candidates to be in bangalore)

Notice- Looking candidates with to be joining within 30 Days

Key Responsibilities:

• Design, implement, and optimize scalable data pipelines using Databricks and Apache Spark.

• Architect data lakes using Delta Lake, ensuring reliable and efficient data storage.

• Manage metadata, security, and lineage through Unity Catalog for governance and compliance.

• Ingest and process streaming data using Apache Kafka and real-time frameworks.

• Collaborate with ML engineers and data scientists on LLM-based AI/GenAI project pipelines.

• Apply CI/CD and DevOps practices to automate data workflows and deployments (e.g., with GitHub Actions, Jenkins, Terraform).

• Optimize query performance and data transformations using advanced SQL.

• Implement and uphold data governance, quality, and access control policies.

• Support production data pipelines and respond to issues and performance bottlenecks.

• Contribute to architectural decisions around data strategy and platform scalability.

Requirements

Required Skills & Experience:

• 5+ years of experience in data engineering roles.

• Proven expertise in Databricks, Delta Lake, and Apache Spark (PySpark preferred).

• Deep understanding of Unity Catalog for fine-grained data governance and lineage tracking.

• Proficiency in SQL for large-scale data manipulation and analysis.

• Hands-on experience with **Kafka for real-time data streaming.

• Solid understanding of CI/CD, infrastructure automation, and DevOps principles.

• Experience contributing to or supporting Generative AI / LLM projects with structured or unstructured data.

• Familiarity with cloud platforms (AWS, Azure, or GCP) and data services.

• Strong problem-solving, debugging, and system design skills.

• Excellent communication and collaboration abilities in cross-functional teams.