Data Engineer
Upstream Security is hiring a Data/Platform Engineer in Herzliya, Israel to design, build, and operate scalable data pipelines and ML platform components on its Iceberg-based automotive data lake. Day to day involves ETL/orchestration, data lake architecture, and MLOps tooling using Python, Spark, Kafka, Presto/Trino, Prefect, Kubernetes, MLflow, and dbt.
- Compensation
- Not disclosed
- Location
- Herzliya, Tel Aviv District
- Work setup
- Hybrid
- Seniority
- Not specified
- Employment
- Full time
- Experience
- 4+ years
- Category
- Data engineering
- Posted
Skills
Must have
- BSc/BA in Computer Science, Engineering or a related field
- At least 4 years of experience designing and building data pipelines, analytical tools and data lakes
- Experience with ETL & orchestration tools (e.g. Airflow, Argo, Prefect)
- Experience with distributed data processing tools (e.g. Spark, Kafka, Presto)
- Experience with Python
Nice to have
- Experience working with open-source products
- Experience working in a containerized environment (e.g. k8s)
- Development experience using a general purpose programming language (Java, Scala, Kotlin, Go, etc.)
Full description
Upstream delivers a cloud-based, AI-powered behavioral intelligence platform for physical AI and smart mobility. Upstream empowers customers with advanced AI-driven cybersecurity solutions, including detection and response (XDR), AI Security, quality and operational intelligence, cyber threat intel, SOC services, and more.
Upstream is looking for an experienced Platform Engineer to join our core platform engineering team, focusing on developing and maintaining our AI-powered automotive data management platform. As a Platform Engineer, you will design, build, and operate scalable, production-grade data and ML platform components, ensuring high reliability and performance.
You will work closely with our Data Engineering and Data Science teams to develop and enhance our Iceberg-based data platform, including a data lake, query engine, and MLOps tools, providing a solid AI-ready foundation for our automotive solutions. These solutions support millions of connected vehicles worldwide, focusing on cybersecurity (XDR), vehicle quality, and operational intelligence. You will work with technologies such as Iceberg, Trino, Prefect, GitHub Actions, Kubernetes, JupyterHub, MLflow, and dbt.
This role is full-time and is Herzliya, Israel-based.
Responsibilities
- Design, build, and maintain scalable data pipelines to ingest and transform batch data on our data lake, enabling analytics and ML with strong data quality, governance, observability, and CI/CD.
- Build and expand our foundational data infrastructure, including our data lake, analytics engine, and batch processing frameworks.
- Create robust infrastructure to enable automated pipelines that will ingest and process data into our analytical platforms, leveraging open-source, cloud-agnostic frameworks and toolsets.
- Develop and maintain our data lake layouts and architectures for efficient data access and advanced analytics.
- Develop and manage orchestration tools, governance tools, data discovery tools, and more.
- Work with other team members of the engineering group, including data architects, data analysts, and data scientists, to provide solutions using a use case-based approach that drives the construction of technical data flows.
- End-to-end ownership mindset with a proactive, production-first approach
Requirements
- BSc/BA in Computer Science, Engineering or a related field
- At least 4 years of experience with designing and building data pipelines, analytical tools and data lakes
- Experience with the data engineering tech stack: ETL & orchestration tools (e.g. Airflow, Argo, Prefect), and distributed data processing tools (e.g Spark, Kafka, Presto)
- Experience with Python is a must
- Experience working with open-source products - Advantage