Lead Data Engineer

Data EngineerExperience : 7 - 15 Years

Employment Type : Full-Time

About the Role :

We are looking for an experienced Data Engineer to design, build and maintain scalable data pipelines and data platforms that enable analytics, reporting, AI/ML and data-driven decision-making. The ideal candidate should have strong experience working with large datasets, cloud technologies, data warehouses and enterprise data integration.

Key Responsibilities :

• Design, develop and maintain scalable and reliable data pipelines.

• Build and optimize ETL/ELT processes to integrate data from multiple sources.

• Extract and transform data from ERP, CRM, databases, APIs, IoT and other enterprise systems.

• Develop high-performance SQL queries, data transformations and stored procedures.

• Design and maintain data warehouses, data lakes and lakehouse architectures.

• Work with structured, semi-structured and unstructured data.

• Implement data pipelines using technologies such as Apache Spark, PySpark, Airflow, Kafka and cloud-native services.

• Develop data ingestion and integration frameworks for batch and real-time data.

• Ensure data quality, accuracy, consistency, security and availability.

• Monitor, troubleshoot and optimize data pipelines and workflows.

• Optimize data processing, storage and query performance.

• Collaborate with Data Scientists, AI/ML Engineers, BI Developers, Software Engineers and business stakeholders.

• Support data requirements for AI/ML models, analytics and business intelligence.

• Implement data governance, metadata management and access-control practices.

• Develop technical documentation for data architecture, pipelines and processes.

• Support migration of legacy data platforms to modern cloud-based data architectures.