Substantial professional experience in senior-level data engineering or data development, including the design and maintenance of production-grade data pipelines.
Strong experience with Microsoft Azure data services and Databricks, including the development and optimization of distributed data processing solutions.
Proficiency in PySpark, Python, and SQL for data transformation, analysis, integration, and automation.
Strong knowledge of ETL/ELT processes and data pipeline architecture, including ingestion, transformation, orchestration, and delivery.
Experience with Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Apache Spark.
Practical knowledge of CI/CD practices and automated deployment workflows.
Experience with Git and GitHub for source control, as well as familiarity with Azure DevOps for versioning and deployment automation.
Demonstrated experience in conceptual, logical, and physical data modeling, with an understanding of performance optimization, scalability, and data governance.
Experience integrating and processing large datasets, maintaining data consistency, and ensuring data quality across complex workflows.
Familiarity with cloud migration projects and the technical considerations involved in secure and reliable data transfers.
Ability to collaborate with business and data science teams to support AI model deployment and cloud-based data solutions.
Strong analytical and problem-solving skills, with the ability to investigate data issues, identify root causes, and implement effective improvements.
Full description
This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Data Developer (AI / Databricks) Sênior based in Brazil.
The Senior Data Developer will design, build, and maintain scalable data engineering solutions that support AI initiatives and business transformation. This role focuses on developing robust data pipelines within Microsoft Azure, leveraging Databricks, PySpark, Python, and SQL to process and integrate large volumes of data. You will help ensure data quality, governance, security, and reliability throughout the data lifecycle while optimizing performance and automating deployment workflows. Working closely with business teams, data scientists, and technical specialists, you will contribute to cloud migration projects and the deployment of AI models in production environments. This opportunity is ideal for an experienced data professional who enjoys solving complex engineering challenges, improving data architecture, and delivering dependable, cloud-native solutions. The position is remote within Brazil, with office attendance required for professionals residing in the Campinas metropolitan region, in accordance with the applicable attendance policy.
Accountabilities
- Design, develop, maintain, and optimize ETL/ELT data pipelines using Microsoft Azure, Databricks, and PySpark.
- Implement automated workflows for data ingestion, transformation, integration, processing, and delivery to downstream systems and applications.
- Process and manage large volumes of customer registration and master data, ensuring data accuracy, consistency, integrity, and availability.
- Develop and maintain CI/CD pipelines to automate deployments, streamline development workflows, and support reliable releases.
- Manage source code, version control, and collaborative development workflows using GitHub and Azure DevOps.
- Design, implement, and optimize conceptual, logical, and physical data models to support scalability, performance, and effective data governance.
- Work with Azure Data Factory (ADF), Azure Synapse Analytics, Azure Data Lake Storage, and Spark to build and evolve cloud-based data solutions.
- Contribute to cloud migration initiatives, ensuring secure data transfer, integrity, quality, and continuity throughout the migration process.
- Collaborate with business stakeholders, data scientists, and engineering teams to integrate data solutions and deploy AI models in cloud environments.
-Establish and maintain data quality controls, including data profiling, validation, consistency checks, and ongoing monitoring.
Data Developer (AI / Databricks) Sênior at Jobgether (Remote) — aijobsearch
Excellent technical documentation and communication skills, with the ability to work collaboratively across multidisciplinary teams.
Preferred qualifications: Experience with AI projects and automated data pipelines; knowledge of data-oriented architectures and governance best practices; involvement in Master Data Management (MDM)
Experience with Azure Machine Learning or similar tools for deploying and managing AI models is an advantage.
- Monitor pipeline performance and delivery indicators, identify anomalies and bottlenecks, and implement improvements to data processing workflows.
- Automate recurring processes and queries to improve operational efficiency, reliability, and maintainability.
- Document data architectures, pipeline designs, technical processes, and engineering best practices to support knowledge sharing and maintainability.
- Contribute to the evolution of data architecture, governance practices, and AI-enabled workflows, applying sound engineering principles across the data lifecycle.
Requirements
- Substantial professional experience in senior-level data engineering or data development, including the design and maintenance of production-grade data pipelines.
- Strong experience with Microsoft Azure data services and Databricks, including the development and optimization of distributed data processing solutions.
- Proficiency in PySpark, Python, and SQL for data transformation, analysis, integration, and automation.
- Strong knowledge of ETL/ELT processes and data pipeline architecture, including ingestion, transformation, orchestration, and delivery.
- Experience with Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Apache Spark.
- Practical knowledge of CI/CD practices and automated deployment workflows.
- Experience with Git and GitHub for source control, as well as familiarity with Azure DevOps for versioning and deployment automation.
- Demonstrated experience in conceptual, logical, and physical data modeling, with an understanding of performance optimization, scalability, and data governance.
- Experience integrating and processing large datasets, maintaining data consistency, and ensuring data quality across complex workflows.
- Familiarity with cloud migration projects and the technical considerations involved in secure and reliable data transfers.
- Ability to collaborate with business and data science teams to support AI model deployment and cloud-based data solutions.
- Strong analytical and problem-solving skills, with the ability to investigate data issues, identify root causes, and implement effective improvements.
- Excellent technical documentation and communication skills, with the ability to work collaboratively across multidisciplinary teams.
- Preferred qualifications: Experience with AI projects and automated data pipelines; knowledge of data-oriented architectures and governance best practices; involvement in Master Data Management (MDM) initiatives; familiarity with Databricks MLflow for model lifecycle management; and experience with Data Lake and Data Warehouse architectures.
- Experience with Azure Machine Learning or similar tools for deploying and managing AI models is an advantage.
Benefits
- Remote work: Home-office arrangement for professionals based in Brazil, subject to the specific office attendance policy applicable to residents of the Campinas metropolitan region.
- Health insurance: Medical coverage.
- Dental insurance: Dental coverage.
- Food and meal allowances: Benefits to support food and meal expenses.
- Childcare assistance: Financial support for eligible childcare expenses.
- Extended parental leave: Enhanced parental leave provisions to support employees and their families.
- Fitness and wellbeing: Partnerships with fitness, health, and wellness platforms, including Wellhub (Gympass) and TotalPass.
- Profit-sharing: Participation in a profit-sharing program (PLR), subject to applicable program terms.
- Life insurance: Life insurance coverage.
- Continuous learning: Access to a dedicated learning platform, including internal professional development resources.
- Online education: Partnerships with online course providers to support technical and professional development.
- Language learning: Access to a language-learning platform.
- Discount program: Access to a club offering discounts and special offers.
- Physical and mental wellbeing: Access to a free online platform dedicated to health, mental wellbeing, and overall wellness.
- Parenting support: Courses focused on pregnancy and responsible parenting.
- Inclusive workplace support: Dedicated health and wellbeing professionals, inclusion specialists, and employee affinity groups. Candidates with disabilities can request appropriate support and accommodations during the recruitment process.
- Compensation details: The original job description does not specify a salary range, bonus amount, or equity compensation. Confirm compensation, benefit eligibility, and applicable terms with the hiring team.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.