Senior Data Engineer - Databricks/Apache Spark

About the Role:

We are seeking an experienced Senior Data Engineer with strong expertise in Databricks, PySpark, Python, and Cloud Data Platforms to design, develop, and optimize scalable data solutions. The ideal candidate will be responsible for building robust data pipelines, implementing modern ETL/ELT frameworks, and enabling data-driven decision-making across the organization.

This role requires deep technical expertise in distributed data processing, cloud-native architectures, and performance optimization for large-scale data workloads.

Key Responsibilities:

• Design, develop, and maintain scalable and reliable data pipelines using Databricks, Apache Spark, and PySpark.

• Build and optimize ETL/ELT workflows, data ingestion frameworks, and transformation processes for structured and unstructured data.

• Develop Python-based automation scripts and data processing applications to streamline data operations.

• Optimize Spark workloads for performance, scalability, cost efficiency, and resource utilization.

• Implement and manage data lakes using Delta Lake and modern data engineering architectures.

• Integrate Databricks solutions with Azure Data Factory (ADF), AWS Glue, APIs, databases, and third-party systems.

• Collaborate with data analysts, data scientists, product teams, and business stakeholders to deliver high-quality data solutions.

• Ensure adherence to data governance, security, monitoring, logging, and compliance standards.

• Implement and maintain CI/CD pipelines, DevOps practices, and automated deployment processes.

• Troubleshoot complex data engineering issues and provide performance tuning recommendations.

• Participate in Agile ceremonies and contribute to technical design discussions and architecture decisions.

Required Skills & Qualifications:

• 6+ years of experience in Data Engineering or related roles.

• Strong hands-on expertise in Databricks, Apache Spark, and PySpark.

• Advanced proficiency in Python and SQL.

• Experience building and maintaining large-scale data pipelines and distributed data processing systems.

• Strong understanding of data warehousing, data modeling, and data lake architectures.

• Experience working with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform (GCP).

• Knowledge of data engineering best practices, including performance optimization, monitoring, and governance.

• Experience with version control systems, CI/CD pipelines, and DevOps methodologies.