Senior Data Engineer - Python/PySpark

Role Overview :

As a Senior Data Engineer, you will be at the forefront of designing and implementing robust data pipelines that serve as the backbone for our clients' analytical capabilities. You will work closely with cross-functional teams, including data scientists, business analysts, and cloud architects, to translate complex business requirements into high-performance technical solutions. Your contribution will directly influence the reliability and accessibility of data, enabling stakeholders to make data-driven decisions with confidence and speed.

Key Responsibilities :

• Architect and maintain scalable ETL/ELT pipelines using Azure Data Factory to ensure seamless data ingestion and transformation across diverse sources.

• Develop high-performance data processing applications using PySpark and Azure Databricks to handle large-scale analytical workloads.

• Implement real-time data modeling and structured streaming solutions to support low-latency reporting and decision-making requirements.

• Leverage Delta Live Tables to build reliable, maintainable, and testable data pipelines that adhere to modern data engineering best practices.

• Manage end-to-end CI/CD workflows using Azure DevOps to ensure code quality, version control, and automated deployment of data solutions.

• Optimize complex SQL queries and database schemas to improve system performance and reduce operational costs for client environments.

Required Skillset :

• Demonstrated expertise in designing and deploying end-to-end data engineering solutions within the Azure ecosystem, specifically utilizing Azure Data Factory, Databricks, and SQL.

• Advanced proficiency in Python and PySpark for building complex data transformation logic and distributed computing tasks.

• Proven ability to implement real-time data modeling and structured streaming architectures to solve business-critical data latency issues.

• Strong experience with Delta Live Tables and modern data lakehouse architectures to ensure data integrity and pipeline efficiency.

• Hands-on experience with Azure DevOps for managing infrastructure as code and streamlining the software development lifecycle.

• Excellent communication skills with the ability to articulate technical concepts to non-technical stakeholders and collaborate effectively in a hybrid work environment.

• A minimum of 6 - 9 years of professional experience in data engineering or a closely related field, with a track record of delivering high-quality technical projects.