Senior Data Engineer
Build and optimize scalable data pipelines and governed lakehouse platforms using Databricks, Spark, and AWS to support analytics and AI in a regulated biotech environment.
- Compensation
- Not disclosed
- Location
- India - Hyderabad
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 7+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Master’s degree in Computer Science, Engineering, Information Technology, Data Science, or a related field and at least 7 years of relevant experience.
- Advanced hands-on experience with Databricks, Apache Spark, PySpark, Spark SQL, Delta Lake, Python, and SQL.
Nice to have
- Experience in biotechnology, pharmaceutical, life sciences, manufacturing, or another regulated industry.
Full description
Career Category
Engineering
##
Job DescriptionRole Description
We are seeking an experienced Senior Data Engineer to lead the design, development, and delivery of scalable enterprise data solutions. This role will build and optimize batch and real-time data pipelines, reusable integration frameworks, and governed data platforms that support analytics, AI, and self-service data access.
The ideal candidate has deep expertise in Databricks, Apache Spark, AWS, data modeling, governance, and production operations. The role also provides technical leadership, defines engineering standards, mentors other engineers, and partners with architecture, business, analytics, data science, and DevOps teams.
Experience in manufacturing, biotechnology, pharmaceutical, life sciences, or another regulated industry is preferred.
Roles and Responsibilities
- Lead the design and development of scalable batch and real-time ETL/ELT pipelines.
- Own complex data solutions from requirements and design through deployment and production support.
- Build cloud-based lakehouse and data-platform solutions using Databricks and AWS.
- Develop reusable, metadata-driven data integration frameworks.
- Integrate structured, semi-structured, and unstructured data from enterprise, manufacturing, API, and third-party sources.
- Optimize Spark workloads, Databricks compute, SQL queries, partitioning, storage, and caching for performance and cost.
- Implement workflow orchestration, monitoring, alerting, data-quality controls, and recovery processes.
- Develop CI/CD pipelines and automated testing for data solutions.
- Implement metadata management, lineage, cataloging, governance, RBAC, and data-security controls.
- Define data models, data contracts, integration patterns, and reusable engineering standards.
- Lead architecture reviews, code reviews, troubleshooting, and root-cause analysis.
- Mentor engineers and provide technical guidance across delivery teams.
- Collaborate with architects, analysts, data scientists, product teams, and DevOps teams.
- Support estimation, sprint planning, technical roadmaps, and delivery-risk management.
- Evaluate emerging technologies and recommend solutions based on scalability, security, maintainability, and business value.
- Participate in operational support, including occasional off-hours support.