Lead Data Engineer (Big Data Specialist)
Lead a distributed data engineering team for a data analytics firm, architecting and maintaining large-scale batch and real-time pipelines and infrastructure using Spark, the Hadoop ecosystem, and Kafka. Covers data warehousing, data lakes, ETL/ELT, data quality, and mentoring junior engineers. Fully remote position.
- Compensation
- Not disclosed
- Location
- Gauteng, South Africa
- Work setup
- Remote
- Seniority
- Lead
- Employment
- Not specified
- Experience
- 5+ years
- Category
- Data engineering
- Posted
Skills
Must have
- Master's or Bachelor's degree in Computer Science, Engineering, or a related quantitative field
- 5+ years of experience in data engineering, with a strong focus on Big Data technologies
- Proven experience designing and implementing data pipelines using frameworks like Apache Spark
- In-depth knowledge of distributed systems and Big Data technologies (Hadoop, Hive, HDFS, Kafka)
- Proficiency in programming languages such as Python, Scala, or Java
- Strong SQL skills and experience with various database technologies (relational and NoSQL)
- Familiarity with data warehousing concepts and data modeling techniques
- Excellent leadership, problem-solving, and communication skills
- Ability to work effectively in a remote, collaborative team environment
Nice to have
- Experience with cloud data platforms (AWS, Azure, GCP)
Full description
About the Role
Our client, a cutting-edge data analytics firm, is seeking a visionary Lead Data Engineer with specialized expertise in Big Data technologies to join their distributed team. This pivotal role involves architecting, building, and maintaining scalable data pipelines and infrastructure that enable advanced analytics and machine learning initiatives. You will lead a team of talented data engineers, drive innovation in data processing, and ensure the reliability and efficiency of our client's data ecosystem. This is a fully remote position, offering the ultimate flexibility to contribute to critical projects from anywhere in the world. If you are passionate about harnessing the power of Big Data and have a proven track record of building robust data solutions, this is an exceptional opportunity to shape the future of data engineering.
Key Responsibilities
- Lead the design, development, and optimization of large-scale batch and real-time data pipelines.
- Build and manage robust, scalable, and fault-tolerant data infrastructure using Big Data technologies (e.g., Spark, Hadoop ecosystem, Kafka).
- Develop and maintain data warehousing solutions and data lakes.
- Implement ETL/ELT processes to ingest, transform, and load data from various sources.
- Ensure data quality, integrity, and security across all data systems.
- Collaborate with data scientists, analysts, and software engineers to understand data needs and deliver effective solutions.
- Mentor and guide junior data engineers, fostering best practices and technical excellence.
- Automate data processes and monitor system performance for optimal efficiency.
- Stay current with emerging Big Data technologies and best practices.
Requirements
- Master's or Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
- 5+ years of experience in data engineering, with a strong focus on Big Data technologies.
- Proven experience designing and implementing data pipelines using frameworks like Apache Spark.
- In-depth knowledge of distributed systems and Big Data technologies (Hadoop, Hive, HDFS, Kafka).
- Proficiency in programming languages such as Python, Scala, or Java.
- Strong SQL skills and experience with various database technologies (relational and NoSQL).
- Experience with cloud data platforms (AWS, Azure, GCP) is highly desirable.
- Familiarity with data warehousing concepts and data modeling techniques.
- Excellent leadership, problem-solving, and communication skills.
- Ability to work effectively in a remote, collaborative team environment.
Benefits
- Highly competitive salary and performance-based bonuses.
- Comprehensive health, dental, and vision insurance.