Contango
Data Engineer
- Compensation
- Not disclosed
- Location
- Abu Dhabi, AE
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Full time
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Nice to have
- MLflow or light MLOps experience (for the data science touchpoints)
- Dbt / dagster / airflow or similar transformation tools
- Understanding of security and compliance (esp. around client data)
- Past experience in consulting or client-facing roles
Full description
Tasks
About the Role
We are an emerging AI-native product-driven agile start-up under Abu Dhabi government AND we are seeking a motivated and technically versatile Data Engineer to join our team. You will play a key role in delivering data platforms pipelines and ML enablement within a Databricks on Azure environment.
As part of a stream-aligned delivery team youll work closely with Data Scientists Architects and Product Managers to build scalable high-quality data solutions for clients. Youll be empowered by a collaborative environment that values continuous learning Agile best practices and technical excellence.
Ideal candidates have strong hands-on experience in Databricks Python ADF and are comfortable in fast-paced client-facing consulting engagements.
Skills and Experience requirements
- Technical
- Databricks (or similar) e.g. Notebooks (Python SQL) Delta Lake job scheduling clusters and workspace management Unity Catalog access control awareness
- Cloud data engineering ideally Azure including storage (e.g. ADLS S3 ADLS) compute and secrets management
- Development languages such as Python SQL C# javascript etc. especially data ingestion cleaning and transformation
- ETL / ELT including structured logging error handling reprocessing strategies APIs flat files databases message queues event streaming event sourcing etc.
- Automated testing (ideally TDD) pairing/mobbing. Trunk Based Development Continuous Deployment and Infrastructure-as-Code (Terraform)
- Git and CI/CD for notebooks data pipelines and deployments
- Integration & Data Handling
- Experienced in delivering platforms for clients including file transfer APIS (REST etc.) SQL/NoSQL/graph databases JSON CSV XML Parquet etc
- Data validation and profiling - assess incoming data quality. Cope with schema drift deduplication and reconciliation
- Testing and monitoring pipelines: Unit tests for transformations data checks and pipeline observability
- Working Style
- Comfortable leveraging the best of lean agile and waterfall approaches. Can contribute to planning estimation and documentation but also collaborative daily re-prioritisation
- Able to explain technical decisions to teammates or clients
- Documents decisions and keeps stakeholders informed
- Comfortable seeking support from other teams for Product Databricks Data architecture
- Happy to collaborate with Data Science team on complex subsystems