Azure Data Engineer - PySpark/Databricks

Company Overview:

Coders Brain Technology & Private Limited is a global technology consulting and staffing firm that specializes in providing high-end digital transformation solutions. The company operates across diverse sectors including banking, healthcare, retail, and manufacturing, helping enterprises modernize their IT infrastructure and data ecosystems. With a focus on agility and technical precision, Coders Brain enables organizations to leverage emerging technologies to solve complex business challenges at scale.

Role Overview:

As a Data Engineer based in Pune, you will be responsible for designing, building, and maintaining robust data pipelines that power our clients' analytical and operational platforms. You will collaborate closely with data architects, business analysts, and cross-functional engineering teams to transform raw data into actionable insights. Your work will directly influence the efficiency of data-driven decision-making processes, ensuring high availability and integrity of data assets across the enterprise.

Key Responsibilities:

• Develop and optimize scalable ETL/ELT pipelines using Azure Data Factory to facilitate seamless data movement across cloud environments.

• Architect and maintain high-performance data processing solutions using Databricks and Delta Lake to support large-scale analytical workloads.

• Design efficient data models and write complex SQL queries to extract, transform, and load data from diverse sources into centralized data warehouses.

• Implement automated deployment workflows using CI/CD practices to ensure reliable and consistent code delivery across development and production environments.

• Integrate external data sources by building and maintaining RESTful API connections and parsing complex JSON structures to enrich internal datasets.

• Manage version control and collaborative development workflows using Git to maintain high standards of code quality and documentation.

Required Skillset:

• Possess 4 to 5 years of professional experience in designing and implementing end-to-end data engineering solutions within cloud-native ecosystems.

• Demonstrate advanced proficiency in Python for data manipulation and automation, alongside expert-level SQL skills for database management.

• Exhibit deep technical expertise in the Azure data stack, specifically Azure Data Factory, Databricks, and Delta Lake, with a focus on performance tuning.

• Show strong capability in handling semi-structured data formats such as JSON and integrating data via RESTful services.

• Maintain a disciplined approach to software development by utilizing Git for version control and implementing CI/CD pipelines to streamline deployment.

• Communicate complex technical concepts effectively to non-technical stakeholders and work collaboratively within a hybrid team environment in Pune.

• Hold a Bachelor's degree in Computer Science, Information Technology, or a related quantitative field, reflecting a strong foundation in data structures and algorithms.