ASSERTIVE BUSINESS SOLUTIONS
Lead Data Engineer - Python/SQL
- Compensation
- Not disclosed
- Location
- Hyderabad
- Work setup
- Not specified
- Seniority
- Lead
- Employment
- Not specified
- Experience
- 8+ years
- Category
- Data engineering
- Posted
Skills
Full description
Job Description:
- 8+ yrs of experience in data engineering with 23 yrs in a lead or architect role.
- Experience in Microsoft Fabric, ADF, DWH, Databricks, PySpark, Snowflake, and Power BI.
- Strong in SQL and Python/Scala.
- Experience with Azure, AWS, and GCP.
- Notice Period: Immediate.
Key Responsibilities :
- Architect and deploy end-to-end data pipelines using Azure Data Factory and Spark to streamline data ingestion and transformation for enterprise-level reporting.
- Design and maintain high-performance Lakehouse architectures utilizing Delta Lake and Azure Data Lake Storage to ensure data integrity and accessibility.
- Lead the migration and optimization of legacy data warehouses into modern Fabric-based environments to enhance query performance and reduce operational latency.
- Implement MLOps best practices by integrating ML Flow into data workflows, enabling seamless model deployment and lifecycle management for data science teams.
- Mentor junior engineers and conduct rigorous code reviews to maintain high standards of software engineering and data quality across all projects.
- Partner with business stakeholders to identify data bottlenecks and implement automated solutions that improve overall data throughput and reliability.