Data engineer (python) in Feature Store
Mid/Mid+ Data Engineer building a Feature Store platform that computes, stores, and serves features for ML models. Day to day: developing production Python pipelines for large datasets with PySpark, Polars, Airflow, ClickHouse and Delta Lake/Iceberg, plus some Go, Docker and Kubernetes. B2B, hybrid from Warsaw, PLN 80-140/h net.
- Compensation
- PLN 80–PLN 140 / hr
- Location
- Warszawa, Poland
- Work setup
- Hybrid
- Seniority
- Lead
- Employment
- Full time
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Very good knowledge of Python
- Experience building, developing, and maintaining production data pipelines
- Practical knowledge of PySpark or another distributed processing tool
- Very good knowledge of SQL
- Experience processing large datasets, including volumes exceeding available memory
- Knowledge of data flow orchestration tools, e.g. Airflow or Argo Workflows
- Practical knowledge of Git and Docker
- Knowledge of good practices for designing and developing data pipelines
- Ability to write readable, testable, maintainable code with attention to code quality and data correctness
- Ability to debug and analyze technical problems
- Independence in completing tasks, accountability, and a problem-solving mindset
- Openness to communicating with platform users and supporting them
- Willingness to continuously develop and learn new technologies
- Ability to communicate in Polish
- Readiness to work in a hybrid model
Nice to have
- Experience with lakehouse formats and technologies, e.g. Delta Lake or Apache Iceberg
- Knowledge of the Polars library
- Experience with analytical databases, e.g. ClickHouse or StarRocks
- Knowledge of Redis or other solutions for fast data serving
- Knowledge of Go
- Experience with Django or building backend services supporting data platforms
- Experience with Feature Store platforms, e.g. Feast, Tecton, or Hopsworks
- Knowledge of MLOps and preparing data for ML models
- Knowledge of distributed processing/task management tools, e.g. Ray or Kubeflow
- Experience with message queues and data streaming, e.g. Kafka
- Experience with Kubernetes and cloud environments, e.g. GCP or Azure
- Experience with CI/CD and automating development processes
- Experience writing unit, integration, and data quality tests
- Experience with monitoring, logging, and observability tools
- Ability to work with AI coding tools (Claude Code, Cursor, Copilot, etc.) and openness to agentic coding
Full description
Описание: We are looking for a Mid / Mid+ Data Engineer to join the team developing a Feature Store platform- a solution responsible for computing, storing, and serving features used by machine learning models. You will work on developing a production data platform, building and optimizing pipelines that process large volumes of information. Projects include retrieving data from analytical warehouses, distributed processing, preparing and storing data, and serving it to systems and ML models. Python is the main programming language, and our daily technologies include PySpark, Polars, Airflow, ClickHouse, Delta Lake, and Apache Iceberg. We also use Go in selected components. This role is for someone with strong Python skills and experience processing large datasets who wants to develop practical, prod
Задачи
- Design, develop, and maintain the Feature Store platform.
- Build and develop data processing pipelines in Python.
- Process large volumes of data using tools such as PySpark and Polars.
- Integrate with data warehouses and analytical systems.
- Prepare and store data in formats used in data lake and lakehouse architectures.
- Orchestrate and automate data flows.
- Develop components and services that support the data platform.
- Prepare and serve features used by ML models, including in online scenarios.
- Optimize the performance, cost, and scalability of data processing workflows.
- Ensure data quality, correctness, and consistency, including validation, schema handling, and process idempotency.
- Debug, analyze errors, and solve technical problems.
- Work with containerization and runtime environments based on Docker and Kubernetes.
- Ensure code quality, testability, and maintainability of solutions.
- Participate in designing data architecture and technical solutions.
- Collaborate with data scientists, ML engineers, backend developers, architects, and product owners.
- Work with platform users, help solve problems, and advise on how to use the available solutions.
- Participate in task estimation, code reviews, and project planning.
- Use AI tools that support programming and automate developers' work.
Требования
- Very good knowledge of Python.
- Experience building, developing, and maintaining production data pipelines.
- Practical knowledge of PySpark or another distributed processing tool.
- Very good knowledge of SQL.
- Experience processing large datasets, including volumes exceeding available memory.
- Knowledge of data flow orchestration tools, e.g. Airflow or Argo Workflows.
- Practical knowledge of Git and Docker.
- Knowledge of good practices for designing and developing data pipelines.