Millennium IT ESP
Senior Engineer – Data Engineering & Analytics
Senior data engineer at an IT services firm who designs, builds, and operates enterprise batch and real-time ETL/ELT pipelines and lakehouse/warehouse solutions on Databricks, Snowflake, Microsoft Fabric, Cloudera, Informatica, and Oracle. Also builds Power BI semantic models, applies governance and DataOps, and works directly with client stakeholders through delivery, UAT, and SLA support.
- Compensation
- Not disclosed
- Location
- Colombo, Sri Lanka
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 2+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Bachelor's Degree in Data Science or higher qualification (e.g., MSc in Data Science, Data Engineering, or AI) from a recognized university
- 2–5 years of professional experience building and operating enterprise data pipelines, data warehouses, or lakehouses
- Hands-on experience with at least two of: Databricks, Snowflake, Microsoft Fabric/Azure Data Services, Cloudera, Informatica (IDMC/PowerCenter), or Oracle (ADW/Exadata/ODI)
- Strong experience with Apache Spark and distributed data processing at scale
- Solid understanding of data modelling, data quality, and data governance principles
- Experience developing Power BI semantic models and reports
- Strong communication skills and ability to work directly with client stakeholders
- Strong SQL and Python (PySpark) skills
Nice to have
- Professional certifications (Databricks Certified Data Engineer, SnowPro Core/Advanced, Microsoft DP-700/DP-600, Informatica IDMC, Oracle Autonomous Database)
- Experience migrating legacy ETL platforms (Informatica PowerCenter, SSIS, ODI) or on-premises data warehouses to cloud lakehouse platforms
- Experience with real-time streaming and CDC tools, including Kafka and Oracle GoldenGate
- Exposure to GenAI data engineering, including RAG pipelines, vector databases, and LLM-ready data preparation
- Experience with dbt, Terraform, Azure DevOps, or GitHub Actions
- Prior experience in banking, telecommunications, or public-sector data projects
- Knowledge of Scala or Java
Full description
Job Description
- Design, build, and maintain scalable batch and real-time ETL/ELT pipelines on enterprise data platforms, including Databricks, Snowflake, Microsoft Fabric, Cloudera, Informatica IDMC, and Oracle
- Develop lakehouse and data warehouse solutions using Medallion (Bronze/Silver/Gold) architecture on Delta Lake, Apache Iceberg, OneLake, Snowflake, and Oracle Autonomous Data Warehouse (ADW)
- Build and orchestrate data workflows using Databricks Lakeflow, Fabric Data Factory, Azure Data Factory, Informatica Cloud Data Integration, Snowflake Streams & Tasks, and Apache Airflow
- Implement Change Data Capture (CDC) and streaming ingestion using Oracle GoldenGate, Apache Kafka, and Spark Structured Streaming
- Apply dimensional data modelling, including Kimball star schemas, to deliver analytics-ready data marts
- Develop Power BI semantic models, including Direct Lake, DAX, and row-level security, in partnership with BI and analytics teams
- Implement data governance, security, data quality, and lineage using Databricks Unity Catalog, Microsoft Purview, Cloudera SDX, and Informatica Data Quality
- Prepare governed, high-quality data for AI and Machine Learning use cases, including feature pipelines and RAG-ready datasets using vector search capabilities on Databricks, Snowflake Cortex, and Oracle AI Vector Search
- Apply DataOps practices, including Git-based version control, CI/CD for data pipelines, automated testing, and Infrastructure as Code
- Monitor, troubleshoot, and optimize production pipelines for performance and cloud cost, supporting the practice's 99.90% uptime SLA commitment
- Work directly with client stakeholders across the delivery lifecycle, including requirements gathering, data model validation, UAT, Go-Live, and post-Go-Live SLA support
- Mentor junior engineers and contribute to internal engineering standards, reusable pipeline frameworks, and technical documentation
Person Specification
- Possess a Bachelor's Degree in Data Science or a higher qualification, such as an MSc in Data Science, Data Engineering, or Artificial Intelligence, from a recognized university
- Have 2–5 years of professional experience in building and operating enterprise data pipelines, data warehouses, or lakehouses
- Possess hands-on experience with at least two of the following platforms: Databricks, Snowflake, Microsoft Fabric/Azure Data Services, Cloudera, Informatica (IDMC/PowerCenter), or Oracle (ADW/Exadata/ODI)
- Demonstrate strong experience with Apache Spark and distributed data processing at scale
- Possess a solid understanding of data modelling, data quality, and data governance principles
- Have experience developing Power BI semantic models and reports
- Demonstrate strong communication skills and the ability to work directly with client stakeholders