Senior Data Engineer (Azure Databricks)

The Asia Data Office is looking for an experienced Senior Data Engineer (Azure Databricks) to join our team to support our regional partners. If you have passion for data and a desire to drive quality data solutions within a data driven culture, then read on to learn more about this opportunity!

We are looking for the right individual who would be interested and enjoy the opportunity to support data operations, strong willingness to learn and build data operations skills through team collaboration, desire to drive and execute on our data strategy, willing to help mature our agile practices, and is looking for an opportunity to embrace technical, operational, and informal leadership roles on the team.

Position Responsibilities:

• Selecting and integrating any Big Data tools and frameworks required providing requested capabilities
• To be able to monitor the performance and advice necessary infrastructure changes or performance tuning in codes.
• Support business users in use of the Enterprise Data Lake (EDL) as part of BAU
• Perform POC on new integration patterns and solutions.
• Maintain and monitor platform stability and performance of EDL
• Attends to Incidents and change requests
• Write and maintain technical documentation
• Perform unit tests and system integration tests
• Executes updates, patches, and other activities required to maintain and enhance the operations of the EDL
• Supports the Agile delivery squads when required

Required Qualifications:

• 8 years of relevant experience in Dat Engineer, Platform Engineer, DevOps Engineer and Software Engineer
• Proficient understanding of distributed computing principles
• Good Knowledge of Hadoop cluster, Azure, with all included services
• Ability to solve any ongoing issues with operating the cluster
• Experience with Azure ADLS, Data Factory, Databricks, Event Hubs
• Experience with data engineering projects
• i.e. integration of data from multiple data sources to a target system
• Good knowledge of data querying tools – Azure Synapse Analytics, Databricks SQL Warehouse
• Knowledge in Python and Spark
• Knowledge in any Change Data Capture (CDC) solutions (e.g. Debezium)
• Experience in working on Agile projects is an advantage.
• Experience on Azure Cloud and/or other Cloud platforms
• Database development and management (e.g. Azure SQL, Cosmos)
• Good knowledge of insurance Data Domains (Policy, Claims, Underwriting, Customer, Finance & Distribution.
• Strong understanding of Data Governance, Regulations and Compliance

Preferred Qualifications:

• Project management experience is an advantage.