Senior Data Engineer - Python/PySpark

Role Overview:

We are seeking a seasoned Senior Data Engineer to join our high-impact data platform team in Pune. In this role, you will architect, build, and maintain scalable data pipelines that process massive datasets, serving as the backbone for our analytical and machine learning initiatives. You will collaborate closely with cross-functional teams, including data scientists, product managers, and business stakeholders, to translate complex requirements into robust technical solutions. By optimizing data architecture and ensuring high data integrity, you will directly influence strategic decision-making and drive significant business outcomes for our global client base.

Key Responsibilities:

• Design and implement high-performance data processing frameworks using PySpark to handle large-scale batch and real-time data ingestion.

• Develop and maintain sophisticated ETL/ELT pipelines in Python to streamline data flow across distributed systems, ensuring reliability and minimal latency for downstream applications.

• Partner with data architects to refine data models and storage strategies, ensuring the infrastructure supports evolving business intelligence needs.

• Mentor junior engineering talent by conducting rigorous code reviews and promoting best practices in software engineering and data governance.

• Troubleshoot and resolve complex performance bottlenecks in production environments to maintain high availability and operational excellence.

Required Skillset:

• Demonstrated expertise in building and optimizing distributed data systems using PySpark and Python, with a deep understanding of memory management and performance tuning.

• Proven ability to communicate complex technical concepts to non-technical stakeholders, ensuring alignment between engineering efforts and business goals.

• Strong analytical mindset with the capacity to solve intricate data challenges in a fast-paced, hybrid work environment based in Pune.

• Ability to work effectively within an agile team structure, fostering collaboration and knowledge sharing across diverse global teams.

• A Bachelors or Masters degree in Computer Science, Engineering, or a related quantitative field, backed by 8 - 10 years of professional experience in data engineering roles.