Careerwise-UK
Data Engineer- Databricks/ SQL/ Python/ PySpark
Data engineer joining a global client's BI & Analytics team in London (hybrid, 3 days/week in the Central London office), building and maintaining data pipelines and warehouse/lakehouse solutions with Azure Databricks, PySpark, SQL, Python, ADF and the broader Azure ecosystem. Salary up to £56k base plus pension and healthcare.
- Compensation
- £56K / yr
- Location
- London (Hybrid: 3 days travel to the...
- Work setup
- Hybrid
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Demonstrable delivery experience using Azure Databricks
- Solid delivery experience using ADF, ADLS Gen2 and the Azure Cloud Ecosystem
- Solid SQL skills — querying syntax, DDL and DML statements
- Solid Python skills for data loading, transformation and exploration in Databricks Notebooks, including Pandas
- Strong understanding of core Databricks/Spark concepts: Delta Live Tables, Delta Sharing, Medallion Architecture, Dataframes, Workflows, Unity Catalogue, UDFs
- Understanding of Data Warehouse design principles: Star Schema, F&D design, Dimension and Fact loading patterns, SCD and CDC
- Experience with MS DevOps for Scrum management, Repos, and CI/CD pipelines between Azure and DevOps
- Proven track record in data engineering developing data pipelines, warehouses and lakehouses for BI, reporting and analytics
Nice to have
- Relevant Azure certifications such as AZ-900, DP-900, DP-203 or DP-500
- Relevant Databricks certifications: Lakehouse Data Engineer Associate or Professional
Full description
Data Engineer-Databricks: SQL/ Python/ PySparkLocation: London (Hybrid: 3 days travel to the office in Central London per week + 2 days’ work from Home)Salary- up to £56K base + pension + healthcare.
Data Engineer required for one of our global clients in the UK for their BI & Analytics Team. You will be working with the leading tools and platforms such as Power Platform, Databricks, ADF.
Duties & Responsibilities
- Data Pipeline development/orchestration/maintenance
- Creating robust, fit-for-purpose data modelling solutions to address multiple business cases
- Root cause analysis and pro-active investigation to identify potential issues and optimize ETL pipelines, notify end-users and propose adequate solutions
- Prepare documentation for further reference
Software Knowledge
- Demonstratable delivery experience using Azure Databricks
- Solid delivery experience using ADF, ADLS Gen2, and the Azure Cloud Ecosystem
- Solid SQL skills – Querying syntax, DDL and DML statements
- Solid Python skills – able to use the Python programming concepts in Databricks Notebooks for data loading, transformation and exploration including familiarity with Pandas
- Strong understanding of core Databricks and Spark concepts such as Delta Live Tables, Delta Sharing, Medallion Architecture, Dataframes, Workflows, Unity Catalogue, Delta Live Tables, and UDF’s
- Understands the key principles of Data Warehouse design including Star Schema, F&D design, Dimension and Fact loading patterns, SCD and CDC concepts
- Experience with use of MS DevOps for Scrum management and with Repos and use of CI / CD Pipelines between Azure and DevOps
- Proven track record in a data engineering role where the focus has been on developing data pipelines, data warehouses, data lakehouses, and similar data repositories for BI, Reporting and Analytics use
Nice to have
- Any relevant Azure certifications / exams such as: AZ-900, DP-900, DP-203, or DP-500
- Any relevant Databricks certifications: Lakehouse Data Engineer Associate, : Lakehouse Data Engineer Professional