eClerx
Process Manager
- Compensation
- Not disclosed
- Location
- Pune, Maharashtra, India
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Full time
- Experience
- 4+ years
- Category
- Not specified
- Posted
Skills
Must have
- 4–8 years of experience in Data Engineering.
- Strong hands-on experience with GCP.
- Strong proficiency in SQL, preferably extensive experience with BigQuery SQL.
- Strong hands-on experience in Python and/or PySpark.
- Experience developing ETL/ELT data pipelines.
- Hands-on experience with BigQuery.
- Hands-on experience with Google Cloud Storage (GCS).
- Experience with Apache Airflow / Cloud Composer.
- Good understanding of data pipeline architecture and data engineering best practices.
- Experience with Git and CI/CD practices.
- Exposure to Jenkins or similar CI/CD tools.
- Experience in production support, monitoring, troubleshooting, and performance optimization.
Nice to have
- Experience working with campaign, marketing, customer, or clickstream data.
- Experience with large-scale data processing.
- Knowledge of Dataflow / Apache Beam.
- Knowledge of Pub/Sub and event-driven architectures.
- Experience with real-time or streaming data pipelines.
- Knowledge of data warehousing and dimensional data modeling.
- Experience with data quality frameworks and validation.
- Knowledge of GCP IAM and security concepts.
- Experience with Cloud Monitoring / Logging.
- Experience working in Agile/Scrum environments.
Full description
Job Description – GCP Data Engineer
Role Overview
We are looking for an experienced GCP Data Engineer to design, develop, and maintain scalable data pipelines and ETL/ELT frameworks on the Google Cloud Platform. The candidate will work extensively with BigQuery, Python/PySpark, Airflow/Cloud Composer, and Google Cloud Storage to process and transform large volumes of campaign, customer, and clickstream data.
The ideal candidate should have strong hands-on experience in data engineering, excellent SQL and Python skills, and a good understanding of cloud-based data pipeline architecture. The role will involve building reliable production pipelines, optimizing BigQuery performance and costs, implementing automation, and supporting critical data workflows.
Key Responsibilities
Data Pipeline Development
- Design, develop, and maintain scalable and reliable data pipelines on GCP.
- Build batch and, where required, near-real-time data ingestion and transformation pipelines.
- Develop robust ETL/ELT frameworks for large volumes of structured and semi-structured data.
- Ingest data from multiple sources into Google Cloud Storage and BigQuery.
- Develop reusable and modular data engineering components.
- Implement appropriate error handling, retry mechanisms, logging, and data validation.
BigQuery & Data Engineering
- Develop complex and optimized SQL queries in BigQuery for data transformation and analysis.
- Optimize BigQuery performance through:
- Partitioning
- Clustering
- Query optimization
- Efficient table design
- Appropriate data types and storage strategies
- Monitor and optimize BigQuery processing costs.
- Design scalable data models suitable for large-scale campaign, customer, and clickstream datasets.
- Troubleshoot data quality, performance, and pipeline-related issues.
Airflow / Cloud Composer
- Develop and maintain DAGs using Apache Airflow / Cloud Composer.
- Implement scheduling, dependency management, retries, failure handling, and alerting.
- Monitor production workflows and proactively resolve failed or delayed jobs.
- Build reusable operators and workflow components where required.
- Ensure critical data pipelines meet agreed SLAs.
Python / PySpark
- Develop data processing and transformation logic using Python and/or PySpark.
- Write clean, reusable, scalable, and production-ready code.