Data Engineer Databricks (Performance & Security)
Data engineer at consulting firm Syffer who optimizes Databricks performance, security, and governance (Unity Catalog, Terraform) on Azure-based data platforms, monitors clusters and jobs with Grafana, and supports data engineers with SQL/PySpark tuning and streaming (Kafka, Structured Streaming).
- Compensation
- Not disclosed
- Location
- Portugal
- Work setup
- Hybrid
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- Deep knowledge of Databricks configuration, optimization, and administration
- Experience with PySpark, SQL, and data performance tuning
- Familiarity with Unity Catalog, DABs, and Terraform integration
- Strong understanding of data governance, cost optimization, and cluster policies
- Fluent in Portuguese and English
Nice to have
- Basic to intermediate experience with Kafka or interest in learning streaming processing
Full description
Data Engineer Databricks (Performance & Security)
Syffer is an all-inclusive consulting company focused on talent, tech and innovation. We exist to elevate companies and humans all around the world, making change, from the inside to the outside.
We believe that technology + human kindness positively impacts every community around the world. Our approach is simple, we see a world without borders, and believe in equal opportunities. We are guided by our core principles of spreading positivity, good energy and promote equality and care for others.
Our hiring process is unique! People are selected by their value, education, talent and personality. We dont present ethnicity, religion, national origin, age, gender, sexual orientation or identity.
Its time to burst the bubble, and we will do it together!
What you’ll do:
- Optimize Databricks performance across jobs, clusters, and data workloads;
- Implement and maintain governance, security, and permissions via Unity Catalog and Terraform;
- Contribute to Databricks feature enablement Delta Live Tables, Databricks Asset Bundles (DABs), and cost efficiency improvements;
- Monitor cluster and job performance using Grafana and internal observability tools;
- Support data engineers in troubleshooting and fine-tuning jobs and SQL queries;
- Explore and progressively implement streaming features (Structured Streaming, Autoloader, Kafka integration);
- Manage Databricks environments integrated with Airflow on AKS, Azure ADLS Gen2, and Terraform;
- Collaborate with DataOps to ensure high availability, logging, and monitoring coverage;
- Implement performance and security benchmarks for production workloads;
- Utilize stack components including Talend, Azure ADLS Gen2, Azure Databricks, Airflow (AKS), Terraform, SQL, Python, PySpark, Grafana, Kafka (Aiven), GitLab, Databricks Asset Bundles (DABs), and DQX (Data Quality Framework);
- Hybrid Work Model.
Who you are:
- Deep knowledge of Databricks configuration, optimization, and administration;
- Experience with PySpark, SQL, and data performance tuning;
- Familiarity with Unity Catalog, DABs, and Terraform integration;
- Strong understanding of data governance, cost optimization, and cluster policies;
- Basic to intermediate experience with Kafka or interest in learning streaming processing;
- Fluent in Portuguese and English;