Senior Data Engineer - SQL/Python

Job Summary :

We are seeking an experienced Senior Data Engineer with strong expertise in Databricks, Apache Airflow, SQL, and Python to design and optimize data pipelines, ETL workflows, and cloud-based data solutions. The ideal candidate will have deep experience in big data processing, cloud-based analytics, and automation to drive efficient and scalable data engineering solutions.

Key Responsibilities :

• Design, develop, and optimize data pipelines using Databricks and Apache Airflow.

• Implement PySpark-based transformations and processing in Databricks for handling large-scale data.

• Develop and maintain SQL-based data pipelines, ensuring performance tuning and optimization.

• Create Python scripts for automation, data transformation, and API-based data ingestion.

• Work with Airflow DAGs to schedule and orchestrate data workflows efficiently.

• Optimize data lake and data warehouse performance for scalability and reliability.

• Integrate data pipelines with cloud platforms (AWS, Azure, or GCP) and various data storage solutions.

• Ensure adherence to data security, governance, and compliance standards.

Required Skills & Qualifications :

• 8-9 years of experience in Data Engineering or related fields.

• Strong expertise in Databricks (PySpark, Delta Lake, DBSQL).

• Proficiency in Apache Airflow for scheduling and orchestrating workflows.

• Advanced SQL skills for data extraction, transformation, and performance tuning.

• Strong programming skills in Python (pandas, NumPy, PySpark, APIs).

• Experience with big data technologies and distributed computing.

• Hands-on experience with cloud platforms (AWS / Azure / GCP).

• Expertise in data warehousing and data modeling concepts.

• Understanding of CI/CD pipelines, version control (Git)

• Experience with data governance, security, and compliance best practices.

• Excellent troubleshooting, debugging, and performance optimization skills.