Data Engineer - Google Cloud Platform/PySpark

Job Summary:

We are seeking a skilled Data Engineer with strong expertise in GCP, PySpark, and Scala to design, develop, and optimize scalable data pipelines and ETL workflows. The ideal candidate will have hands-on experience working with modern data platforms, orchestration tools, and cloud-native data processing frameworks to support large-scale data engineering initiatives.

Key Responsibilities:

• Design, develop, deploy, monitor, and optimize ETL workflows for batch data processing.

• Build and maintain scalable data pipelines using Scala, PySpark, Airflow, BigQuery, Dataproc, and GCP services.

• Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) with robust data quality checks and data summarization processes.

• Ensure high-performance and reliable data processing through continuous optimization and tuning.

• Develop observability frameworks, monitoring solutions, and operational runbooks to support production systems.

• Manage workflow orchestration using Apache Airflow, including dependency management, scheduling, and idempotent execution.

• Collaborate with cross-functional teams to understand business requirements and translate them into scalable data solutions.

• Troubleshoot and resolve data pipeline issues while ensuring data integrity and availability.

• Follow best practices for data engineering, coding standards, and operational excellence.

• Contribute to data governance initiatives and ensure compliance with organizational data management standards.

Required Skills:

• 6+ years of experience in Data Engineering.

• Strong hands-on experience with Google Cloud Platform (GCP).

• Proficiency in PySpark and Scala for large-scale data processing.

• Experience with BigQuery, Dataproc, and Apache Airflow.

• Solid understanding of ETL development, data warehousing, and batch processing frameworks.

• Experience implementing Medallion Architecture and data quality frameworks.

• Knowledge of performance optimization, monitoring, observability, and operational support processes.