Senior Data Engineer - Big Data & AI Infrastructure

About the Role

Our client, a dynamic innovator in AI and emerging technologies, is seeking a highly experienced Senior Data Engineer to build and maintain robust big data infrastructure that powers our advanced AI initiatives. This role is integral to ensuring the seamless flow, storage, and accessibility of massive datasets required for machine learning and analytics. You will be responsible for designing, developing, and optimizing data pipelines and data warehousing solutions. This position is based in Randburg, Gauteng, ZA and offers a flexible hybrid work model, combining the benefits of in-office collaboration with remote flexibility.

Key Responsibilities

• Design, build, and maintain scalable and reliable data pipelines for ingesting, processing, and transforming large volumes of data from various sources.
• Develop and manage data warehousing solutions and data lakes that support AI/ML workloads and business intelligence.
• Optimize data infrastructure for performance, cost-efficiency, and reliability.
• Implement data governance, security, and quality standards.
• Collaborate with data scientists and AI engineers to understand their data needs and provide efficient data solutions.
• Monitor data systems, troubleshoot issues, and implement solutions to ensure high availability.
• Evaluate and integrate new data technologies and tools to enhance our data platform.
• Develop and maintain comprehensive documentation for data pipelines and infrastructure.
• Mentor junior data engineers and contribute to the team's technical growth.

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field.
• 5+ years of professional experience in data engineering, with a strong focus on big data technologies.
• Proven expertise in designing and implementing ETL/ELT pipelines using tools like Apache Spark, Kafka, Flink, or similar.
• Extensive experience with cloud data platforms (e.g., AWS S3, Redshift, Glue; Azure Data Lake, Synapse; GCP BigQuery, Dataflow).
• Proficiency in SQL and experience with NoSQL databases.
• Strong programming skills in Python or Java/Scala.
• Familiarity with data warehousing concepts and data modeling techniques.
• Experience with containerization technologies (Docker, Kubernetes) is a plus.
• Excellent problem-solving skills and the ability to work effectively in a hybrid team environment.
• Strong communication and collaboration abilities.

Benefits

Our client offers a competitive salary, attractive bonuses, and comprehensive benefits including health insurance, retirement plans, and paid time off. Employees benefit from a flexible hybrid work schedule, allowing for a good work-life balance. Access to continuous learning and professional development resources, including training and conferences, is provided. Join a collaborative and innovative environment focused on leveraging data to drive AI advancements.