Senior Data Engineer – Python & PySpark

Senior Data Engineer – Python & PySpark

📄 Job Summary

We are seeking an experienced Senior Data Engineer with strong expertise in Python, PySpark, SQL, and Big Data technologies.

The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and ETL/ELT workflows for processing large volumes of structured and unstructured data. The role requires hands-on experience with distributed data processing, cloud platforms, orchestration tools, and performance optimization of big data applications.

🛠️ Key Responsibilities

📌 Data Pipeline Development

• Design, develop, and maintain scalable data pipelines using:

• Python

• Apache Spark / PySpark

• Build reusable and efficient data processing frameworks.

📌 ETL / ELT Development

• Develop and optimize ETL/ELT workflows for:

• Data ingestion

• Data transformation

• Data processing

• Process large volumes of structured and unstructured data.

📌 Big Data Processing

• Work with big data technologies such as:

• Hadoop ecosystem

• Hive

• Spark

• Implement distributed computing solutions for high-performance processing.

📌 Data Modeling & Warehousing

• Support:

• Data modeling

• Data architecture

• Data warehousing solutions

• Ensure scalability and maintainability of data systems.

📌 SQL & Database Management

• Write and optimize:

• Complex SQL queries

• Data transformation logic

• Work with:

• Relational databases

• Non-relational databases

📌 Cloud & Orchestration

• Deploy and manage data solutions on cloud platforms such as:

• AWS

• Azure

• GCP

• Work with orchestration tools like:

• Apache Airflow

📌 Data Quality & Governance

• Perform:

• Data validation

• Data cleansing

• Data transformation