Cloud Data Engineer – Lakehouse Pipeline Development
Builds and maintains scalable cloud ETL/ELT pipelines and lakehouse architectures that turn large datasets into analytics-ready assets. Core stack: SQL, Python/PySpark, Spark, one of AWS/Azure/GCP, table formats like Delta Lake/Iceberg/Hudi, and orchestration tools such as Airflow.
Compensation
Not disclosed
Location
Dubai, United Arab Emirates
Work setup
Hybrid
Seniority
Junior
Employment
Contract
Experience
2+ years
Category
Data engineering
Posted
Skills
accessibility
agile
airflow
analytics
anomaly-detection
api
automation
avro
aws
azure
business-intelligence
ci-cd
cloud
cloud-native
data-analytics
data-engineering
data-governance
data-ingestion
data-lake
data-modeling
data-pipelines
data-quality
data-warehousing
delta-lake
dimensional-modeling
distributed-computing
elt
etl
event-driven-architecture
gcp
git
iam
iceberg
integration-testing
json
lakehouse
machine-learning
networking
nosql
parquet
pyspark
python
rdbms
rest
scala
spark
sql
test-automation
unit-testing
version-control
workflow-orchestration
Must have
Bachelor's degree in Computer Science, IT, Software Engineering, Data Engineering, or related field
Practical programming experience with Python, PySpark, Scala, or a similar data engineering language
Understanding of data lake, data warehouse, and lakehouse architectures
Experience developing and maintaining ETL/ELT pipelines for large-scale data processing
Knowledge of cloud data services on AWS, Azure, or GCP
Practical exposure to at least one major cloud platform (AWS, Azure, or GCP)
Familiarity with Apache Spark and distributed computing concepts
Understanding of Delta Lake, Apache Iceberg, Apache Hudi, or comparable lakehouse table formats
Experience with relational databases and familiarity with NoSQL or modern data stores
Knowledge of data modelling techniques (dimensional modelling, star schemas, slowly changing dimensions)
Understanding of file formats such as Parquet, Avro, JSON, and CSV
Familiarity with workflow orchestration tools and automated scheduling
Knowledge of Git, source control, and code review practices
Understanding of REST APIs, data ingestion patterns, and event-driven data processing
Familiarity with unit testing, integration testing, data validation, and automated deployment
Awareness of data privacy, information security, access management, and governance standards
Strong analytical and problem-solving abilities
Nice to have
2–5 years of relevant experience in data engineering, cloud data platforms, or ETL/ELT development
Master's degree in a relevant field
Professional certifications in cloud computing, data engineering, or data analytics
Experience with data quality checks, monitoring, logging, and pipeline recovery mechanisms
Understanding of performance optimization, scalable data storage, and cloud resource management
Experience with Agile environments, version control, and CI/CD tools
Full description
1. Job Summary
The Elite Job is seeking a skilled and detail-oriented Cloud Data Engineer – Lakehouse Pipeline Development to design, develop, implement, and maintain scalable cloud-based data pipelines and modern lakehouse architectures. This role is ideal for a technology professional who enjoys working with large datasets, building reliable data processing solutions, and transforming complex data into high-quality, analytics-ready assets. The successful candidate will be responsible for developing robust ETL/ELT pipelines, integrating data from multiple sources, and implementing efficient data storage and processing solutions across cloud platforms. The role involves working with data lakes, data warehouses, distributed processing frameworks, and lakehouse technologies to support business intelligence, advanced analytics, machine learning, and data-driven decision-making. The Cloud Data Engineer will collaborate closely with data architects, analytics engineers, data scientists, software developers, and business stakeholders to ensure that data solutions meet performance, security, reliability, and business requirements. The ideal candidate combines strong programming skills with an understanding of cloud infrastructure, data modelling, data governance, and modern engineering practices. This position offers an opportunity to contribute to innovative data engineering initiatives, improve enterprise data capabilities, and develop expertise in next-generation cloud data platforms.
2. Key Responsibilities
Lakehouse Architecture Development: Design, develop, and maintain scalable lakehouse solutions that combine the flexibility of data lakes with the reliability and management capabilities of data warehouses.
Data Pipeline Engineering: Build, deploy, and optimize batch and streaming data pipelines to ingest, transform, validate, and deliver structured, semi-structured, and unstructured data.
ETL/ELT Development: Implement efficient extraction, transformation, and loading processes using modern data engineering frameworks, reusable components, and automated workflows.
Cloud Platform Integration: Develop data solutions using cloud services such as AWS, Microsoft Azure, or Google Cloud Platform, depending on project requirements.
Distributed Data Processing: Use technologies such as Apache Spark, PySpark, SQL, and related processing frameworks to handle large-scale datasets efficiently.
Lakehouse Table Management: Work with open table formats and storage technologies such as Delta Lake, Apache Iceberg, or Apache Hudi to support reliable data updates, schema evolution, and transaction management.
Data Source Integration: Connect data pipelines to databases, APIs, enterprise applications, event streams, file systems, and third-party data sources.
Cloud Data Engineer – Lakehouse Pipeline Development at The Elite Job (Dubai, United Arab Emirates) — aijobsearch
Excellent attention to detail and commitment to data quality
Ability to translate business requirements into technical data solutions
Effective written and verbal communication skills
Demonstrated experience building, deploying, and maintaining production data pipelines
Data Modelling and Transformation: Develop efficient data models, transformation logic, and curated datasets that support reporting, analytics, and downstream applications.
Data Quality Assurance: Implement validation rules, reconciliation checks, schema enforcement, anomaly detection, and monitoring mechanisms to maintain data accuracy and consistency.
Pipeline Orchestration: Schedule, coordinate, and monitor data workflows using tools such as Apache Airflow, cloud-native orchestration services, or equivalent platforms.
Performance Optimization: Identify bottlenecks in data processing, storage, and query execution, and optimize partitioning, file sizes, caching, and resource utilization.
Security and Governance: Apply appropriate access controls, encryption, data classification, auditing, and governance practices to protect sensitive information.
Monitoring and Troubleshooting: Investigate pipeline failures, resolve data processing issues, manage dependencies, and implement preventive measures to improve operational reliability.
CI/CD and Automation: Establish automated testing, deployment pipelines, infrastructure configuration, and version control practices to support consistent and repeatable releases.
Documentation: Maintain technical documentation covering pipeline architecture, data flows, transformation logic, dependencies, operational procedures, and troubleshooting guidelines.
Cross-Functional Collaboration: Work with data analysts, business intelligence teams, data scientists, and engineering stakeholders to understand requirements and deliver dependable data products.
Continuous Improvement: Evaluate emerging cloud data technologies, engineering methodologies, and lakehouse capabilities to improve scalability, efficiency, maintainability, and cost effectiveness.
3. Required Skills and Qualifications
Educational Qualifications
Bachelors degree in Computer Science, Information Technology, Software Engineering, Data Engineering, or a related technical discipline.
A masters degree in a relevant field is an advantage but is not mandatory.
Relevant professional certifications in cloud computing, data engineering, or data analytics are beneficial.
Technical Skills
Strong proficiency in SQL, including complex queries, joins, aggregations, window functions, and query optimization.
Practical programming experience with Python, PySpark, Scala, or another suitable data engineering language.
Understanding of data lake, data warehouse, and lakehouse architectures.
Experience developing and maintaining ETL/ELT pipelines for large-scale data processing.
Knowledge of cloud data services available on AWS, Microsoft Azure, or Google Cloud Platform.
Familiarity with Apache Spark and distributed computing concepts.
Understanding of Delta Lake, Apache Iceberg, Apache Hudi, or comparable lakehouse technologies.
Experience with relational databases and familiarity with NoSQL databases or other modern data stores.
Knowledge of data modelling techniques, including dimensional modelling, star schemas, and slowly changing dimensions.
Understanding of file formats such as Parquet, Avro, JSON, and CSV.
Familiarity with workflow orchestration tools and automated scheduling.
Knowledge of Git, source control, code reviews, and collaborative development practices.
Understanding of REST APIs, data ingestion patterns, and event-driven data processing.
Familiarity with unit testing, integration testing, data validation, and automated deployment practices.
Awareness of data privacy, information security, access management, and governance standards.
Professional Skills
Strong analytical and problem-solving abilities.
Excellent attention to detail and a commitment to data quality.
Ability to translate business requirements into technical data solutions.
Effective written and verbal communication skills.
Ability to work independently and collaborate with multidisciplinary teams.
Strong organizational skills and the ability to manage competing priorities.
Willingness to learn new technologies and adapt to evolving data engineering practices.
4. Experience
Typically, 2–5 years of relevant experience in data engineering, cloud data platforms, ETL/ELT development, or large-scale data processing is preferred.
Demonstrated experience building, deploying, and maintaining production data pipelines.
Practical exposure to at least one major cloud platform, such as AWS, Azure, or Google Cloud.
Experience with Python, SQL, and distributed data processing tools.
Familiarity with lakehouse architecture and modern table formats is highly desirable.
Experience implementing data quality checks, monitoring, logging, and pipeline recovery mechanisms.
Understanding of performance optimization, scalable data storage, and cloud resource management.
Experience working in Agile development environments and using version control and CI/CD tools is an advantage.
Candidates with strong foundational knowledge, relevant projects, internships, or demonstrable practical experience may also be considered for junior-level opportunities, depending on the specific vacancy requirements.
5. Working Hours
Employment Type: Full-time.
Work Arrangement: Remote, hybrid, or office-based, depending on the position and business requirements.
Typical Schedule: Approximately 40 hours per week, subject to the applicable employment agreement.
Working Days: Generally Monday to Friday, with schedules determined by the assigned team and location.
Collaboration: Occasional flexibility may be required to coordinate with colleagues or stakeholders across different time zones.
Project Deadlines: Some additional flexibility may be necessary during major deployments, production incidents, or critical project milestones.
The final working schedule, location requirements, and employment conditions will be communicated during the recruitment process.
6. Knowledge, Skills and Abilities
Data Engineering Knowledge
The candidate should understand the principles of data ingestion, transformation, storage, processing, orchestration, and consumption. They should be able to design reliable data workflows that accommodate growing data volumes while maintaining data integrity and operational efficiency.
Cloud Computing Expertise
The candidate should understand cloud storage, compute resources, identity and access management, networking fundamentals, monitoring, and resource optimization. The ability to select appropriate cloud services based on technical and business requirements is important.
Lakehouse Development Capabilities
The candidate should be able to contribute to lakehouse implementation, including raw and curated data layers, schema management, incremental processing, data versioning, and reliable data access. Familiarity with concepts such as ACID transactions, schema evolution, and time travel in supported table formats is beneficial.
Programming and Automation
The candidate should be capable of writing readable, maintainable, and reusable code. They should understand error handling, logging, testing, modular development, and automation practices that improve the reliability of data pipelines.
Analytical Thinking
The candidate must be able to investigate data discrepancies, identify root causes, interpret processing failures, and recommend effective solutions. Strong reasoning skills are essential when working with complex datasets and interconnected systems.
Communication and Collaboration
The ability to explain technical concepts clearly, document solutions, participate in design discussions, and coordinate with cross-functional teams is important for successful project delivery.
Security and Compliance Awareness
The candidate should understand secure data handling, least-privilege access, encryption, auditability, and responsible data management. They should follow applicable organizational policies and relevant regulatory requirements.
Adaptability and Continuous Learning
The candidate should demonstrate a willingness to explore new tools, improve existing systems, share technical knowledge, and respond constructively to changing business priorities.
7. Benefits
Competitive Compensation: Salary packages aligned with experience, technical expertise, responsibilities, and market conditions.
Flexible Working Options: Remote or hybrid working arrangements where available and approved for the role.
Professional Development: Opportunities to strengthen skills in cloud engineering, distributed computing, data architecture, and lakehouse technologies.
Certification Support: Potential access to approved training programmes and relevant professional certification resources.
Career Advancement: Opportunities to progress into senior data engineering, cloud architecture, platform engineering, or technical leadership roles.
Collaborative Environment: Opportunities to work with technology professionals across data, software, analytics, and cloud infrastructure teams.
Technical Learning: Exposure to modern engineering tools, scalable architectures, and emerging data platform capabilities.
Performance Recognition: Opportunities for recognition based on contribution, quality of delivery, collaboration, and achievement of agreed objectives.
Work-Life Balance: Support for sustainable working practices, clear priorities, and effective workload management.
Employee Benefits: Any applicable paid leave, insurance, retirement contributions, or other benefits will depend on the employment contract, location, and company policy.
Specific benefits, compensation, and eligibility conditions will be confirmed by the hiring team during the recruitment process.
8. Why Join The Elite Job?
Joining The Elite Job as a Cloud Data Engineer – Lakehouse Pipeline Development offers an opportunity to contribute to modern data engineering solutions and build a career in a rapidly evolving technology field.
Work on Modern Data Technologies: Apply your skills to cloud-native data pipelines, distributed processing, lakehouse architectures, and enterprise data integration.
Solve Meaningful Technical Challenges: Help develop reliable and scalable data solutions that improve information accessibility, support analytics, and enable better business decisions.
Expand Your Technical Expertise: Strengthen your knowledge of cloud platforms, automation, data quality, data governance, and performance optimization.
Collaborate Across Teams: Work with professionals who bring different perspectives to data architecture, analytics, software engineering, and business problem-solving.
Build a Long-Term Career: Develop the technical and professional capabilities required to pursue future opportunities in senior data engineering, cloud architecture, data platform development, and technical leadership.
Contribute to Engineering Excellence: Help establish reusable pipeline frameworks, reliable operational processes, and maintainable data solutions that can evolve with business needs.
If you enjoy solving complex data problems, building dependable systems, and continuously improving your engineering capabilities, this role may be an excellent next step in your professional journey.