Lead Data Engineer
6-month contract (Inside IR35) Lead Data Engineer, hybrid 2 days/week in Edinburgh/Glasgow, placed by Harvey Nash with a public-sector client. Day to day: designing AWS data warehouse/lake solutions, building ETL/ELT pipelines with Python, SQL and Spark, and creating serverless, event-driven analytics in 2-week Agile sprints.
- Compensation
- Not disclosed
- Location
- City of Edinburgh, United Kingdom
- Work setup
- Not specified
- Seniority
- Lead
- Employment
- Contract
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Strong experience with AWS data and analytics services (Redshift, Glue, S3, Athena, Lambda, Step Functions, RDS/Aurora, DynamoDB, DMS, EMR, CloudWatch, IAM)
- Strong programming experience in Python and SQL
- Experience with Spark (PySpark preferred) and large-scale data processing
- Experience developing ETL/ELT pipelines and data warehouse solutions
- Knowledge of API integration and microservices architectures
- Experience working in Agile/Scrum delivery environments
- Must hold, or be happy to apply for, a valid Basic Disclosure Scotland
Nice to have
- Experience creating dashboards, reporting and monitoring with QuickSight, Power BI, Grafana, ELK
Full description
Lead Data Engineer | 6 Month Contract | (Inside IR35) | Hybrid, 2 days pw Edinburgh/Glasgow | Starting ASAP
Day Rate: £DOE
About the role:
Harvey Nash's Pub Sec client is seeking an experienced AWS Data Engineer to design, develop, and support scalable data warehouse and cloud-based data solutions. The successful candidate will work within an Agile Scrum team, delivering end-to-end data engineering solutions, ETL pipelines, and analytics platforms that support business intelligence and digital applications.
You will:
- Design, build, and maintain cloud-based data warehouse and data lake solutions on AWS.
- Develop and support ETL/ELT processes for data ingestion, transformation, and integration.
- Build and optimise data pipelines using Python, SQL, Spark, and AWS services.
- Develop serverless and event-driven solutions using AWS Lambda, Step Functions, EventBridge, SQS, and SNS.
- Support and enhance existing data platforms, applications, and integrations.
- Work closely with business and technical stakeholders to deliver solutions in 2-week Agile sprints.
- Create and maintain dashboards, reporting, and monitoring solutions using QuickSight, Power BI, Grafana, and ELK.
- Implement data quality, security, governance, and operational monitoring best practices.
Essential Criteria
Strong experience with AWS data and analytics services including:
- Redshift
- Glue
- S3
- Athena
- Lambda
- Step Functions
- RDS/Aurora
- DynamoDB
- DMS
- EMR
- CloudWatch
- IAM
Strong programming experience in Python and SQL.
Experience with Spark (PySpark preferred) and large-scale data processing.
Experience developing ETL/ELT pipelines and data warehouse solutions.
Knowledge of API integration and microservices architectures.
Experience working in Agile/Scrum delivery environments.
This role has been deemed Inside IR35 by the client. Applicants must hold, or be happy to apply for, a valid Basic Disclosure Scotland.