Azure Data Engineer - Python/SQL

We are looking for an experienced Azure Data Engineer with 8 to 12 years of experience in designing, developing, and maintaining scalable data solutions on Microsoft Azure. The ideal candidate should have strong expertise in Azure Databricks, Azure Data Factory, ADLS, SQL, Python/PySpark, and ETL processes, along with an understanding of ERP data and financial data environments.

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows on Azure.

• Develop data transformation and processing solutions using Azure Databricks, Python, PySpark, and SQL.

• Build and optimize Databricks notebooks for data transformation, cleansing, aggregation, and table creation.

• Develop and maintain data ingestion pipelines using Azure Data Factory (ADF).

• Work with Azure Data Lake Storage (ADLS) for scalable storage and management of structured and unstructured data.

• Integrate data from multiple sources, including ERP systems, databases, APIs, and enterprise applications.

• Understand ERP data structures, business processes, and data flows and translate them into scalable data solutions.

• Implement data quality, validation, reconciliation, and error-handling processes across data pipelines.

• Optimize data pipelines and queries for performance, scalability, reliability, and cost efficiency.

• Work with Alteryx and Power BI to support data analysis, reporting, and visualization requirements.

• Collaborate with business, analytics, finance, risk, and technology teams to understand requirements and deliver data solutions.

• Support data governance, security, audit, and compliance requirements.

• Troubleshoot data pipeline and production issues and perform root-cause analysis.

• Create technical documentation covering data pipelines, transformations, data flows, and operational procedures.

• Contribute to cloud modernization and data platform improvement initiatives.

Required Skills & Experience :

• 8 to 12 years of experience in Data Engineering, ETL, or related data technologies.

• Strong hands-on experience with Microsoft Azure data services.

• Strong expertise in Azure Databricks and Databricks Notebooks.

• Proficiency in SQL and strong programming experience in Python and/or PySpark.

• Hands-on experience with Azure Data Factory (ADF).

• Strong experience with Azure Data Lake Storage (ADLS).

• Strong understanding of ETL/ELT concepts, data integration, data transformation, and data warehousing.

• Experience designing and implementing data pipelines using multiple data sources.

• Good understanding of Azure cloud services, security, networking, and data platform architecture.

• Experience working with ERP data and ERP data flows.

• Hands-on experience with Alteryx and Power BI for data analysis and visualization.

• Strong understanding of data quality, reconciliation, validation, and performance optimization.

• Good analytical, problem-solving, and stakeholder-management skills.