Real-Time Data Engineer: Kafka, Spark & Cloud Platforms
Design and implement real-time data streaming solutions using Apache Kafka and Spark Structured Streaming, with Python/Scala development on Databricks in a cloud environment (Azure or AWS). Day-to-day work involves building ingestion pipelines, managing messaging systems like Kafka/MSK, and troubleshooting Spark workloads.
- Compensation
- Not disclosed
- Location
- Mexico
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Experience designing and implementing high-velocity data streaming solutions with Apache Kafka
- Real-time processing experience with Spark Structured Streaming
- Ability to troubleshoot Spark workloads
- Proficiency in Python or Scala on Databricks
- Experience working in a cloud environment (Azure or AWS)
- Experience building data ingestion pipelines
- Experience deploying data platforms on Azure or AWS
- Experience managing messaging systems such as Kafka/MSK
- Experience handling data formats
Full description
Aptonet is seeking an experienced data engineering professional to design and implement high-velocity data streaming solutions using Apache Kafka and Spark Streaming. You will develop real-time processing with Spark Structured Streaming, troubleshoot Spark workloads, and work with Python/Scala on Databricks in a cloud environment.
Responsibilities include building ingestion pipelines, deploying data platforms on Azure or AWS, managing messaging systems like Kafka/MSK, and handling data formats