ZScale
Data Engineer
- Compensation
- Not disclosed
- Location
- Remote
- Work setup
- Remote
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Core skills: Azure Cloud, Databricks workflows, DLT (Delta Live Table), Notebooks, Python, SQL, PySpark, ADLS Gen2 ADF, GitHub, PL/SQL, ORACLE, ETL experience, understanding of Delta Lake architecture
- Ability to design and orchestrate data pipelines using Databricks Workflows and DLT, and a strong understanding of Medallion Architecture.
- Expertise in developing Databricks notebooks for scalable solutions using Python, SQL, and PySpark.
- Understanding of Delta Lake architecture, CDC patterns, and Lakehouse.
- Strong understanding of Delta table key features like ACID-compliant Delta tables, time travel, schema enforcement, and deep and shallow clones.
- Performance tuning using Liquid clustering, partitioning, Z-ordering, and data skipping in delta tables.
- Knowledge on data governance (Unity Catalogue), data security (RBAC, fine-grained access control), and data sharing (Delta Sharing).
- Proficient in working with Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory (ADF), and Terraform for provisioning and management of Azure resources.
- Knowledge of dealing with Spark streaming and auto-loaders in Databricks.
- Strong experience in analysing and understanding legacy Informatica ETL workflows, including mappings, transformations, and data flow logic, to support seamless migration to Databricks-based data pipe
- Hands-on experience in implementing CI/CD pipelines using Jenkins to automate the deployment of Databricks notebooks, jobs, and data workflows.
- Integrating GitHub with Databricks Repos to enable seamless code synchronisation, change tracking, and automated deployment workflows.
- knowledge of Snowflake, Oracle, MySQL, and Shell scripting for diverse data integration.
- Knowledge of Power BI and Azure Synapse Analytics for data analytics dashboards and reports.
Full description
Requirements:
- Core skills: Azure Cloud, Databricks workflows, DLT (Delta Live Table), Notebooks, Python, SQL, PySpark, ADLS Gen2 ADF, GitHub, PL/SQL, ORACLE, ETL experience, understanding of Delta Lake architecture, CDC patterns, Informatica, and Lakehouse.
- Ability to design and orchestrate data pipelines using Databricks Workflows and DLT, and a strong understanding of Medallion Architecture.
- Expertise in developing Databricks notebooks for scalable solutions using Python, SQL, and PySpark.
- Understanding of Delta Lake architecture, CDC patterns, and Lakehouse.
- Strong understanding of Delta table key features like ACID-compliant Delta tables, time travel, schema enforcement, and deep and shallow clones.
- Performance tuning using Liquid clustering, partitioning, Z-ordering, and data skipping in delta tables.
- Knowledge on data governance (Unity Catalogue), data security (RBAC, fine-grained access control), and data sharing (Delta Sharing).
- Proficient in working with Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory (ADF), and Terraform for provisioning and management of Azure resources.
- Knowledge of dealing with Spark streaming and auto-loaders in Databricks.
- Strong experience in analysing and understanding legacy Informatica ETL workflows, including mappings, transformations, and data flow logic, to support seamless migration to Databricks-based data pipelines.
- Hands-on experience in implementing CI/CD pipelines using Jenkins to automate the deployment of Databricks notebooks, jobs, and data workflows.
- Integrating GitHub with Databricks Repos to enable seamless code synchronisation, change tracking, and automated deployment workflows.
- knowledge of Snowflake, Oracle, MySQL, and Shell scripting for diverse data integration.
- Knowledge of Power BI and Azure Synapse Analytics for data analytics dashboards and reports.
Skills:
- Cloud Platform: Azure (ADLS Gen2 ADF) Terraform.
- Data Platform: Databricks (Workflows, DLT, Notebooks).
- Languages/Frameworks: Python, SQL, PySpark, PL/SQL, Shell Scripting.
- Architecture: Delta Lake, Medallion Architecture, Lakehouse ETL/ELT.
- DevOps/CI/CD: GitHub, Jenkins, Databricks Repos.
- Data Governance: Unity Catalogue, Delta Sharing.
- Databases: Oracle, ADLS Gen2 Snowflake, MySQL.
- Analytics: Azure Synapse Analytics, Power BI.