Millennium IT ESP
Engineer - Data Engineering & Analytics
A data engineer in Colombo builds and operates real-time ETL/ELT pipelines and analytics-ready data marts on enterprise platforms like Databricks, Snowflake, Microsoft Fabric, Cloudera, Informatica, and Oracle, covering streaming/CDC ingestion, Kimball modelling, governance, DataOps/CI-CD, and direct client-facing delivery support.
- Compensation
- Not disclosed
- Location
- Colombo, Sri Lanka
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- 2+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Bachelor's Degree in Data Science or higher (e.g. MSc in Data Science, Data Engineering, or AI) from a recognised university
- 2–3 years of professional experience building and operating enterprise data pipelines, data warehouses, or lakehouses
- Hands-on experience with at least two of: Databricks, Snowflake, Microsoft Fabric/Azure Data Services, Cloudera, Informatica (IDMC/PowerCenter), Oracle (ADW/Exadata/ODI)
- Strong experience with Apache Spark and distributed data processing at scale
- Solid understanding of data modelling, data quality, and data governance principles
- Strong communication skills and ability to work directly with client stakeholders
Nice to have
- Professional certifications such as Databricks Certified Data Engineer, SnowPro Core/Advanced: Data Engineer, Microsoft DP-700/DP-600, Informatica IDMC, or Oracle Autonomous Database
- Experience migrating legacy ETL platforms (Informatica PowerCenter, SSIS, ODI) or on-premises data warehouses to modern cloud lakehouse platforms
- Strong SQL and Python (PySpark) skills, along with knowledge of Scala or Java
Full description
Job Description
- Implement real-time ETL/ELT pipelines on enterprise data platforms, including Databricks, Snowflake, Microsoft Fabric, Cloudera, Informatica IDMC, or Oracle
- Implement data workflows using technologies such as Databricks Lakeflow, Fabric Data Factory, Azure Data Factory, Informatica Cloud Data Integration, Snowflake Streams & Tasks, and Apache Airflow
- Implement Change Data Capture (CDC) and streaming ingestion using one or more technologies, including Oracle GoldenGate, Apache Kafka, and Spark Structured Streaming
- Apply dimensional data modelling, including Kimball star schemas, to deliver analytics-ready data marts
- Implement data governance, security, data quality, and lineage using Databricks Unity Catalog, Microsoft Purview, Cloudera SDX, and Informatica Data Quality
- Apply DataOps practices, including Git-based version control, CI/CD for data pipelines, automated testing, and Infrastructure as Code
- Monitor, troubleshoot, and optimise production pipelines for performance and cloud cost efficiency, supporting the practice's 99.90% uptime SLA commitment
- Work directly with client stakeholders throughout the delivery lifecycle, including requirements gathering, data model validation, User Acceptance Testing (UAT), Go-Live, and post-Go-Live SLA support
Person Specification
- Possess a Bachelor's Degree in Data Science or a higher qualification, such as an MSc in Data Science, Data Engineering, or Artificial Intelligence, from a recognised university
- Have 2–3 years of professional experience in building and operating enterprise data pipelines, data warehouses, or lakehouses
- Possess hands-on experience with at least two of the following platforms: Databricks, Snowflake, Microsoft Fabric/Azure Data Services, Cloudera, Informatica (IDMC/PowerCenter), or Oracle (ADW/Exadata/ODI)
- Demonstrate strong experience with Apache Spark and distributed data processing at scale
- Possess a solid understanding of data modelling, data quality, and data governance principles
- Demonstrate strong communication skills and the ability to work directly with client stakeholders
- Professional certifications such as Databricks Certified Data Engineer (Associate/Professional), SnowPro Core or SnowPro Advanced: Data Engineer, Microsoft Certified: Fabric Data Engineer Associate (DP-700) or Fabric Analytics Engineer Associate (DP-600), Informatica IDMC, or Oracle Autonomous Database certifications will be considered an added advantage
- Experience in migrating legacy ETL platforms, including Informatica PowerCenter, SSIS, or ODI, or on-premises data warehouses to modern cloud lakehouse platforms will be considered an added advantage
- Strong SQL and Python (PySpark) skills, along with knowledge of Scala or Java, will be considered an added advantage