Data Engineering Manager - Python/PySpark

Data Engineer ManagerExperience : 14 - 20 Years

Job Description :

We are looking for an experienced Data Engineering Manager to lead data engineering teams and drive the development of scalable cloud-based data solutions.

The ideal candidate should have strong hands-on experience with Python, PySpark, GCP, SQL, ETL and modern data engineering frameworks, along with experience managing large engineering teams.

The role will involve technical leadership, architecture, delivery ownership and team management.

Role & Responsibilities :

• Lead and manage large Data Engineering teams, including teams of 20+ members.

• Design and implement scalable data engineering solutions using Python, PySpark and GCP.

• Lead the development of enterprise-grade ETL pipelines and data processing frameworks.

• Work with Google BigQuery, Cloud Run, Airflow, Dataproc, Dataflow and Composer.

• Design and manage data pipelines using modern data engineering frameworks.

• Work with Kafka and APIs for real-time data ingestion and processing.

• Develop scalable batch and streaming data solutions.

• Ensure data pipelines are reliable, scalable, secure and optimized for performance.

• Provide technical direction and mentorship to data engineering teams.

• Drive engineering best practices around development, testing, deployment and monitoring.

• Collaborate with architects, product managers and business stakeholders to define technical solutions.

• Own delivery timelines, technical quality and overall team performance.

• Identify opportunities to improve data platforms, processes and engineering efficiency.

Required Skills & Experience :

• 14 - 20 years of overall experience with strong experience in Data Engineering.

• Strong hands-on experience with Python and PySpark.

• Strong experience with GCP and cloud-based data engineering.

• Strong SQL skills.

• Experience with ETL and data engineering frameworks.

• Hands-on experience with :

• Google BigQuery

• Cloud Run

• Airflow

• Dataproc

• Dataflow

• Cloud Composer

• Experience with Kafka and API-based data ingestion.

• Strong understanding of batch and streaming data processing.

• Proven experience managing and mentoring large Data Engineering teams (20+ members).

• Strong stakeholder management and communication skills.

• Strong problem-solving and technical leadership abilities.