Senior Data Engineer
Senior data engineer in Kuala Lumpur who designs, builds, and maintains Databricks-based data pipelines, lakehouse, and enterprise data warehouse solutions using PySpark, Delta Lake, Unity Catalog, and CI/CD with GitHub Actions, plus a rotating production support roster.
- Compensation
- Not disclosed
- Location
- Kuala Lumpur, Malaysia
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 4+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Bachelor's Degree in Computer Science, IT, Engineering, Data Science, or related field
- 5-8 years of experience in Data Engineering, Data Warehousing, or Big Data technologies
- 4+ years of hands-on Databricks experience in enterprise environments
- Experience designing enterprise Data Warehouse and Lakehouse architectures
- Strong PySpark development and optimization skills
- Delta Lake and Delta Tables
- Unity Catalog
- Databricks Workflows
- RBAC and ABAC implementation within Unity Catalog
- GitHub and Git-based development workflows
- CI/CD implementation using GitHub Actions or equivalent
- SQL and advanced query optimization
- Cloud platforms (Azure, AWS, or GCP)
- Data security, governance, and compliance frameworks
- Enterprise Data Warehouse architecture and implementation
- Dimensional data modeling (Star Schema and Snowflake Schema)
- Fact and Dimension modeling with Slowly Changing Dimensions (SCD Type 1 & 2)
- Data Warehouse performance tuning and optimization
- Lakehouse architecture and Medallion design patterns
- Willingness to participate in production support and on-call rotation schedules
Nice to have
- Data observability and monitoring
- Infrastructure-as-Code (Terraform preferred)
- Databricks Certified Data Engineer Associate or Professional certification
- Experience implementing enterprise data governance frameworks
- Experience working with regulated industries and compliance requirements
- Knowledge of Data Mesh and modern data platform architectures
- Experience integrating Databricks with Power BI, Tableau, or other BI and analytics platforms
Full description
Design, develop, and maintain scalable data pipelines and data products using Databricks and PySpark.
Build and optimize ETL/ELT solutions supporting batch and near real-time data processing.
Design, develop, and maintain enterprise Data Warehouse and Lakehouse solutions that support reporting, analytics, and AI/ML use cases.
Develop and maintain enterprise data models using Delta Lake and Delta Tables.
Design and implement dimensional data models, including Fact and Dimension tables, Star Schema, Snowflake Schema, and Slowly Changing Dimensions (SCD).
Ensure data quality, reliability, scalability, and performance across the data platform.
Implement best practices for code management, testing, deployment, and operational monitoring.
Databricks Platform & Governance
Implement and manage Unity Catalog for centralized governance, data discovery, and security.
Design and maintain governance frameworks utilizing:
Role-Based Access Control (RBAC)
Attribute-Based Access Control (ABAC)
Fine-grained data permissions
Data lineage and auditing
Configure and manage Delta Sharing to support secure external and internal data collaboration.
Support adoption and administration of Databricks Genie, including:
Security and access governance controls
Performance Optimization
Perform advanced PySpark performance tuning and troubleshooting.
Optimize query performance, cluster utilization, partitioning strategies, and workload management.
Identify bottlenecks and proactively improve platform efficiency and cost optimization.
Optimize Data Warehouse and Lakehouse workloads to support high-performance reporting and analytical processing.
DevOps & Automation
Design and implement CI/CD pipelines for Databricks solutions.
Integrate Databricks development lifecycle with GitHub, GitHub Actions, and enterprise DevOps processes.
Automate deployment, testing, code validation, and release management processes.
Establish infrastructure and data engineering best practices.
Stakeholder Management
Engage with business users, data consumers, architects, analysts, and technology leadership to gather requirements and deliver data solutions.