Azure Data Engineer - Python/SQL

About the Role :

We are looking for an experienced Azure Data Engineer with 7 - 9 years of hands-on experience in designing, developing, and maintaining scalable data engineering solutions on Microsoft Azure.

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines and data engineering solutions using Microsoft Azure.

• Develop and manage ETL/ELT workflows for structured, semi-structured, and unstructured data.

• Build data ingestion pipelines using Azure Data Factory and other Azure data integration services.

• Develop data transformation and processing solutions using Azure Databricks and Apache Spark.

• Write optimized SQL queries, stored procedures, views, and other database objects.

• Integrate data from various sources including relational databases, APIs, files, cloud platforms, and enterprise applications.

• Design and maintain data lakes and data warehouse solutions on Azure.

• Implement batch and near-real-time data processing pipelines as required.

• Develop reusable frameworks and components for data ingestion, transformation, validation, and monitoring.

• Implement data quality checks, validation rules, reconciliation processes, and error handling mechanisms.

• Optimize data pipelines, Spark jobs, SQL queries, and storage for improved performance and cost efficiency.

• Monitor pipeline execution, troubleshoot failures, and resolve production data issues.

• Implement logging, alerting, monitoring, and operational support mechanisms.

• Work closely with data architects, analysts, application teams, and business stakeholders to understand data requirements.

• Participate in data migration, modernization, and cloud transformation initiatives.

• Implement security controls, access management, encryption, and governance requirements for Azure data platforms.

• Maintain technical documentation for data pipelines, architecture, processes, and operational procedures.

• Follow best practices for coding, version control, testing, deployment, and production support.

Technical Skills :

• 7 - 9 years of experience in data engineering, ETL development, or related data platform roles.

• Strong hands-on experience with Microsoft Azure data services.

• Strong experience with Azure Data Factory (ADF) for data integration and pipeline development.

• Hands-on experience with Azure Databricks and Apache Spark.