Data Engineer - Python / Databricks
Hands-on Data Warehouse Python Developer in Dublin: enhance, optimise, support, and modernise an enterprise data warehouse, migrating it from on-prem SQL Server to a cloud Lakehouse using Azure Databricks, Microsoft Fabric, and Python, including ETL/ELT pipelines, AutoSys orchestration, and production/on-call support.
- Compensation
- Not disclosed
- Location
- Dublin, Ireland
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Minimum 10 years' experience in Data Warehouse development and SQL Server development
- Proven track record delivering enterprise-scale Data Warehouse and data engineering solutions
- Extensive hands-on experience as a Data Warehouse Developer and Python Developer
- Experience supporting large-scale production environments and business-critical systems
- Experience delivering Data Warehouse modernisation and cloud migration initiatives
- Strong hands-on experience with Azure Databricks
- Strong experience designing and implementing solutions using Microsoft Fabric
- Advanced SQL Server and T-SQL development skills
- Strong Python programming and automation skills
- Strong cloud data engineering and data warehousing experience
- Experience designing and implementing Lakehouse architectures
- Experience with Medallion (Bronze, Silver, Gold) data models
- Experience with AutoSys workload automation, JIL scripting, job scheduling, and dependency management
- Strong experience in ETL/ELT design and development
- Strong data modelling and database design skills
- Experience troubleshooting and tuning complex SQL and data processing workloads
- Knowledge of CI/CD, DevOps, and automation best practices
Nice to have
- Degree in Computer Science, Software Engineering, Information Systems, Data Science, or related discipline
- Postgraduate qualification in a relevant technology discipline
- Experience within Financial Services, Banking, or Capital Markets
- Experience with Power BI and enterprise analytics platforms
- Experience with Azure Data Factory
- Microsoft Fabric, Azure, Databricks, or related cloud certifications
- Ability to work independently and take ownership of technical delivery
- Ability to balance project delivery, production support, and strategic initiatives
Full description
Role Summary
The role is for a hands-on Data Warehouse Python Developer within the Data Warehouse team. The individual will be responsible for the enhancement, optimisation, support, and modernisation of the Enterprise Data Warehouse platform. The role will focus on designing and delivering cloud-based data solutions using Azure Databricks, Microsoft Fabric, SQL Server, and Python while supporting the migration of the existing on-premise SQL DWH to a modern cloud-based Lakehouse architecture.
The role includes responsibility for production support, operational stability, automation, and continuous improvement of business-critical data platforms. The Enterprise Data Warehouse supports critical regulatory, finance, risk, operational, and management reporting requirements across the organisation.
The successful candidate will work closely with business stakeholders, business analysts, developers, infrastructure teams, and support teams to deliver scalable and reliable data solutions. The individual will be expected to actively participate in Agile delivery activities, production support, modernisation initiatives, and cloud transformation programmes. The role requires a strong technical background combined with the ability to understand business requirements and translate them into robust technical solutions.
Key Interfaces
- Data Warehouse Development Team
- Business Stakeholders
- Business Analysts
- Infrastructure and Cloud Engineering Teams
- Production Support Teams
- DBA Teams
- Architecture and Security Teams
- Risk, Finance, Operations, and Regulatory Reporting Functions
The Successful Candidate Will
- Design, develop, and support enterprise data solutions using Azure Databricks and Microsoft Fabric.
- Build, enhance, and optimise cloud-based ETL/ELT pipelines and data integration processes.
- Develop and maintain SQL Server databases, stored procedures, views, and performance-tuned solutions.
- Develop Python-based solutions for data engineering, automation, monitoring, orchestration, and data quality management.
- Drive the modernisation of the Enterprise Data Warehouse from traditional SQL Server-based architecture to a cloud-native Lakehouse architecture.
- Implement and support Medallion (Bronze, Silver, Gold) data models and best practices.
- Design, maintain, and support AutoSys job schedules, JIL scripting, workflow orchestration, and automation solutions.
- Participate in production support and on-call support activities for business-critical systems.
- Investigate, troubleshoot, and resolve production incidents and performance issues.
- Continuously improve operational processes, automation, monitoring, and platform performance.