Senior Big Data Engineer (GCP / Python / PySpark /Streaming)

Skills

• airflow
• api
• big-data
• bigquery
• ci-cd
• cloud
• data-engineering
• data-modeling
• data-pipelines
• data-quality
• dataproc
• event-driven-architecture
• gcp
• gitlab
• kafka
• nosql
• pyspark
• python
• rest
• scala
• spark
• sql
• workflow-orchestration

Must have

1. Bachelor's degree in Computer Science, Engineering, Information Technology, or equivalent practical experience.
2. 6+ years of experience in Data Engineering, Big Data Engineering, or related disciplines.
3. Strong experience with Python for data engineering and pipeline development.
4. Hands-on experience with Apache Airflow for workflow orchestration (required).
5. Advanced experience with PySpark, Spark SQL, and distributed data processing.
6. Experience with Scala in data engineering environments.
7. Strong knowledge of relational and NoSQL database technologies.
8. Experience writing and optimizing complex SQL queries.
9. Hands-on experience with Apache Kafka and event-driven architectures.
10. Experience building and supporting real-time data pipelines using Spark Streaming.
11. Strong experience with Google Cloud Platform (GCP), specifically:
12. BigQuery
13. Dataproc
14. Experience integrating RESTful APIs into data pipelines and data workflows.
15. Experience implementing CI/CD practices using GitLab CI.
16. Strong understanding of data modeling, data quality, scalability, and performance optimization.
17. Excellent verbal communication and technical problem-solving skills.
18. Ability to explain design decisions and implementation approaches in a collaborative setting.

Full description