Asia Group Functions Lead Data Engineer
Lead Data Engineer at insurer Manulife/John Hancock in Hong Kong, leading and mentoring a data engineering team while building, optimizing, and maintaining enterprise data pipelines and ETL on Azure (Data Factory, Databricks, PySpark), with a focus on data architecture, quality, and governance for the company's digital transformation.
- Compensation
- Not disclosed
- Location
- Hong Kong, Hong Kong
- Work setup
- Not specified
- Seniority
- Lead
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Bachelor's degree in Computer Science, Information Technology, or a related field (Master's is a plus)
- 10+ years of experience as a Data Engineer implementing data pipelines, scheduling, monitoring, notification, and ETL with Azure Data Factory, Databricks, Python/PySpark, Java, Scala
- Understanding of Azure infrastructure: subscriptions, resource groups, RBAC, Azure AD integrations, security principles, networking (VNet, Subnet, NSG, private endpoints), and key/credential managemen
- Experience deploying and integrating Azure Data Services (ADF, Databricks) using DevOps tools: GitHub repository, Jenkins CI/CD pipelines, integrated unit tests
- Knowledge of Azure Data Lake Storage (ADLS Gen2) topology: blob storage, file hierarchy, storage accounts, containers, folders
- Proficient in data mart fact and dimension design, ETL/ELT upsert and type-2 logic with Python/PySpark/stored proc/SQL in ADLS (Databricks lakehouse) or Azure Synapse (Dedicated SQL pool)
Nice to have
- Experience with LLM and AI tools
- Experience using Unity Catalog
- Experience using Power BI with ADLS (Delta Lake), Databricks SQL, Azure Synapse, semantic layer/data model design, and workspace management
- Solid understanding of data privacy and compliance regulations and standard methodologies
- Excellent problem-solving skills and ability to solve technical issues efficiently
- Effective communication skills to collaborate with technical and non-technical partners
- Thorough approach with commitment to delivering high-quality work in a fast-paced environment
Full description
Manulife is embarking on replacing and building new data capabilities to help fuel our bold ambition to become a digital customer leader! We are seeking a skilled and motivated Lead Data Engineer to join our diverse team and play a key role in implementing, optimizing, and maintaining assets that deliver these capabilities! The ideal candidate possesses a solid background in data engineering, ETL processes, and data integration, with a passion for understanding data to drive strategic business decisions.
Asia Group Functions Lead Data Engineer Position Responsibilities:
- Leadership: Mentor and coach data engineers on the team, providing technical guidance and leadership to build a high-performing data engineering team. Use excellent leadership and communication skills to foster a collaborative and high-performing team culture. Ensure that team members are meeting project timelines and delivering high-quality solutions that meet the needs of the business.
- Domain Knowledge: Stay up-to-date with the latest tools and technologies and identify opportunities to enhance Manulife's data infrastructure.
- Teamwork: Partner with other lead data engineers, data analysts, business collaborators, and data scientists to understand data requirements and translate them into effective data engineering solutions.
- Data Pipeline Development: Design, develop, and manage data pipelines that facilitate the detailed extraction, transformation, and loading of data from diverse sources. Leverage expertise in big data and cloud computing to ensure solutions are scalable, efficient, and effective.
- Data Mapping & Integration: Collaborate closely with multi-functional teams to understand and design schemas for data from various source systems and other transactional or application databases, ensuring accuracy and reliability.
- ETL Optimization: Continuously improve and optimize ETL processes to enhance data flow efficiency, minimize latency, and support real-time and batch processing requirements.
- Data Transformation: Implement data cleansing, enrichment, and transformation processes to ensure high-quality data is available for analysis and reporting.
- Data Quality Assurance: Design testing plans, develop and implement data quality checks, validation rules, and supervising mechanisms to maintain data accuracy and integrity.
- Platform Improvement: Collaborate with various technical resources from across the organization to identify and implement improvements to the infrastructure, integrations, and functionality.
- Data Architecture: Work closely with business leads and data architects to design, implement, and lead end-to-end architecture based on business requirements.
- User Documentation: Develop and maintain detailed documentation for data pipelines, processes, and configurations to support seamless onboarding.
- Performance Monitoring: Monitor data pipeline performance and solve issues to ensure efficient data flow and proactively find opportunities for improvement.