Senior Data Engineer (Azure Databricks)
6-month contract-to-hire via Talpro, deployed at a Bengaluru client site: design and build metadata-driven batch and streaming data pipelines on Azure Databricks and Spark, tune performance, ensure data quality, and implement CI/CD. Hybrid (3 days/week in office), UK hours (~2-7 PM IST), up to ₹1.5L/month, immediate joiners only.
- Compensation
- ₹150K / mo
- Location
- Bangalore South, India
- Work setup
- Hybrid
- Seniority
- Senior
- Employment
- Contract
- Experience
- 3+ years
- Category
- Data engineering
- Posted
Skills
Must have
- 6-8 years of overall Data Engineering experience, preferably building data solutions on Microsoft Azure
- At least 5-6 years hands-on Databricks experience, including 2+ end-to-end Databricks projects delivered from concept to production
- 3-4 years of experience designing Databricks Data Platform, familiar with non-functional requirements and data platform design best practices
- Strong understanding of Spark internals, performance tuning, and best practices for building data lakes/warehouses on Databricks
- Hands-on experience building data pipelines using Azure Data Factory and Azure Databricks
- Proficient in Python (including PySpark) for ETL/ELT data integration workflows
- Databricks Certified Data Engineer Associate (at least one Databricks certification required)
- Experience designing and delivering solutions on the Azure Data Analytics stack (Azure Data Lake, Azure Synapse Analytics)
- Background building data warehouse solutions using ETL/ELT tools such as Informatica or Talend
- Strong data management practices: profiling, cleansing, transformation, quality assurance, standardization, deduplication, and matching
- Immediate joiners only
Nice to have
- Microsoft Azure Data Engineer Associate certification (highly desired)
- DevOps experience with Azure DevOps, Chef, Puppet, or Terraform; knowledge of CI/CD pipelines for data projects
- Exposure to structured and unstructured data, including geospatial data or large imaging datasets
- Familiarity with NoSQL and graph databases (MongoDB, Cassandra, Neo4j) and data integration/migration with them
- Experience with streaming/event ingestion platforms such as Azure Event Hub, Azure IoT Hub, Apache Kafka, or Apache NiFi
Full description
Job Title: Senior Data Engineer (Azure Databricks)
**Role Type: Fixed Term Direct Contract with TalproContract Duration: 6 months (Fixed Term Contract)
Years of Experience: 6+ Yrs.Max CTC Offered: INR 1.50 L Per Months
Notice Period: Immediate Joiners Only
Work Mode: Hybrid (3 Days from Office Weekly)
Location: Manyata Tech Park, Bengaluru - Looking for Bengaluru Based Candidates only
Domain: Any IT Domain **
About the Role
Talpro is seeking a Senior Data Engineer (Azure Databricks) for an internal position to be deployed at a client location. This is a 6-month contract-to-hire role – if you excel during the contract, there is potential to transition into a permanent position. You will be on Talpro’s payroll and work with the client’s team in Bangalore. The work model is hybrid (mix of on-site and remote as needed) and aligned to UK business hours (approximately 2 PM to 7 PM IST).
Key Responsibilities
- Data Pipeline Development: Design and implement highly performant, metadata-driven data ingestion and transformation pipelines from multiple sources using Azure Databricks and Apache Spark. This includes building both streaming and batch data processes to handle large-scale data in real time and in bulk.
- Framework & Optimization: Develop scalable and reusable frameworks for data ingestion and transformation of large datasets. Perform Spark performance tuning and optimization to ensure data pipelines are efficient and cost-effective. Provide technical guidance on complex data problems (including handling geospatial data) using Spark DataFrames and other big data tools.
- Data Quality & Integration: Design and implement data quality checks and processes to ensure data integrity is maintained through the pipeline. Integrate end-to-end data pipelines from source systems to target data repositories (data lakes/data warehouses), ensuring consistency, quality, and reliability of data at all times.
- Collaboration & Tooling: Work closely with cross-functional teams (data analysts, data scientists, application developers) to support additional project components such as reporting tools, APIs, and search applications. Evaluate various tools and technologies (e.g., Azure Data Factory, Azure Databricks features, etc.) against project requirements to recommend the best fit.
- Agile Delivery: Operate within an Agile/DevOps delivery environment. Participate in iterative sprint planning, deliver proof-of-concepts (POCs), and guide the project from concept to production. Implement continuous integration/continuous deployment (CI/CD) practices for data pipelines and ensure solutions are delivered in a timely, iterative manner.