Data Engineer

About AllCloud

AllCloud is a leader in amplifying organizations’ cloud potential through AI. With a track record of hundreds of successful migrations and implementations across AWS and Salesforce, AllCloud has developed strategies and solutions that enable businesses of all sizes to remain at the forefront of innovation.

AllCloud is a leader in AI-led professional and managed services. As an AWS Premier and audited managed services Partner, and Salesforce Consulting partner, AllCloud provides comprehensive AI-led cloud journey support, from initial migration to ongoing management through our Engage Managed Services. Our expertise ensures that clients remain aligned with ecosystem best practices while focusing on their core business growth.

AllCloud serves clients across the globe with offices in EMEA and North America.

Job Summary

Integration into the development team for a core DWH project and various AI projects. Responsibilities include the ingestion of large volumes of new data, followed by deep understanding and analysis of data in close collaboration with Data Scientists. Design, development, and optimization of critical, diverse, and large-scale data pipelines – in both cloud and on-premise environments.

Key Responsibilities:

• Take responsibility for the ingestion of large volumes of new data.
• Conduct deep understanding and examination of data in close collaboration with Data Scientists.
• Design and develop critical, diverse, and large-scale data pipelines and processes.
• Deploy and optimize data solutions across both cloud and on-premise environments.

Requirements

• B.Sc. in Computer Science, Engineering, Mathematics, or Statistics - Mandatory.
• 3+ years of experience as a Data Engineer.
• 3+ years of development experience in Python - Mandatory.
• Hands-on experience with Object-Oriented Programming (OOP).
• Hands-on experience with Spark for large-scale data processing - Mandatory.
• Hands-on experience with dbt.
• Experience building and maintaining data pipelines using tools like Airflow or Kubeflow.
• Practical experience and understanding of Docker, and container platforms such as ECS / Kubernetes.
• Understanding of optimization techniques and working with advanced data formats, emphasizing Parquet and Iceberg (as well as Avro, HDF5, Delta Lake).
• Hands-on experience with AWS data services (including Athena, Glue, EMR / EMR Serverless, Redshift, RDS) - Significant advantage.
• Experience working with and integrating BI tools such as QuickSight, Tableau, and PowerBI.
• Understanding of Machine Learning concepts and processes.
• Practical knowledge of GenAI / prompt engineering solutions - Advantage.