Senior Software Engineer, Apple Data Platform
Senior engineer on Apple's Data Platform team advancing Apache DataFusion Comet, the open-source accelerator for Apache Spark that Apple created. Day to day you build native query operators on DataFusion/Arrow, extend Spark and Iceberg/Parquet integration, benchmark and tune performance, and contribute work upstream to Apache open-source communities.
- Compensation
- Not disclosed
- Location
- Cupertino
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 3+ years
- Category
- Data engineering
- Posted
Skills
Must have
- 3+ years of professional software engineering experience
- Proven track record of building distributed systems
- Contributor or committer to Apache Spark and/or Apache DataFusion, Comet
- BS, MS, or PhD in Computer Science or a related field, or equivalent practical experience
Nice to have
- Experience with columnar or vectorized query execution, or with performance profiling and optimization of query engines
- Good knowledge of Apache Iceberg and Apache Parquet internals
- Experience working on or building Spark connectors to external data sources
- Knowledge of Kubernetes, or other compute platforms
Full description
We are seeking exceptional software engineers to work on Apache DataFusion Comet, Apache Spark, and related open-source technologies. Comet, the native accelerator for Apache Spark, was created at Apple and donated to the Apache Software Foundation. If you love pushing distributed query engines to their performance limits without compromising correctness or stability, this role is for you. We are looking for engineers with deep expertise in Spark internals and in related data formats and query engines, such as Apache Iceberg, Parquet, Arrow, and DataFusion. In this role, you'll collaborate with teams across Apple operating Spark at massive scale and actively contribute upstream to these open-source communities.
As an engineer on our team, you will advance Apache Spark to meet Apple's scale while driving significant efficiency gains through Apache DataFusion Comet. You will design and implement native operators and expressions using Apache DataFusion and Apache Arrow, extend Comet's integration across Spark versions, and accelerate Apache Iceberg and Parquet reads and writes, all while guaranteeing strict behavioral and numerical compatibility with vanilla Spark. Working alongside a team of open-source experts, including Apache Spark, DataFusion, and Iceberg committers, you will actively contribute that work back upstream. You will also profile and benchmark production workloads, optimizing memory, shuffle, and I/O to deploy Spark and Comet reliably across on-premises data centers and cloud environments. Your contributions will directly enhance the efficiency, resilience, and scale of systems powering services used by millions of people worldwide.
Minimum Qualifications
- 3+ years of professional software engineering experience.
- A proven track record of building distributed systems.
- Contributor or committer to Apache Spark and/or Apache DataFusion, Comet.
- BS, MS, or PhD in Computer Science or a related field, or equivalent practical experience.
Preferred Qualifications
- Experience with columnar or vectorized query execution, or with performance profiling and optimization of query engines.
- Good knowledge of Apache Iceberg and Apache Parquet internals.
- Experience working on or building Spark connectors to external data sources.
- Knowledge of Kubernetes, or other compute platforms.