Data Engineer - AWS/Databricks

About the Role :

We are looking for an experienced AWS Data Engineer with strong hands-on expertise in cloud data engineering, Databricks, PySpark, SQL, and modern data integration technologies. The ideal candidate should have experience building scalable data pipelines on AWS, working with Databricks and DBT, integrating APIs, and supporting data engineering projects across the complete project lifecycle.

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines and data processing solutions using AWS services.

• Develop and optimize ETL/ELT pipelines using AWS Glue, PySpark, Databricks, and DBT.

• Build data ingestion and processing workflows using AWS S3, Glue, Lambda, EMR, and other AWS data services.

• Develop PySpark jobs for large-scale data transformation and processing.

• Work with Databricks for data engineering, transformation, processing, and pipeline development.

• Develop and optimize SQL queries for data extraction, transformation, validation, and analysis.

• Build and maintain data pipelines using Apache Airflow for workflow orchestration and scheduling.

• Develop serverless data processing solutions using AWS Lambda.

• Design and manage data storage and data ingestion solutions using Amazon S3.

• Work with AWS API Gateway to develop and manage API-based data integrations.

• Develop and integrate REST and SOAP APIs with data engineering workflows.

• Implement reliable data ingestion from internal and external systems through APIs, files, databases, and other sources.

• Develop DBT models and transformation workflows to create reliable and maintainable data transformation layers.

• Implement data validation, quality checks, error handling, logging, and monitoring across data pipelines.

• Troubleshoot pipeline failures, data quality issues, performance problems, and integration issues.

• Optimize data processing jobs and pipelines for performance, scalability, reliability, and cost efficiency.

• Collaborate with data architects, business analysts, developers, QA teams, and other technical stakeholders.

• Interact directly with clients to understand requirements, clarify technical details, provide project updates, and resolve data-related issues.

Tech Stack :

• AWS Data Services (S3, Glue, Lambda, EMR, API Gateway)

• PySpark, SQL, DBT, Apache Airflow

• Databricks, REST/SOAP APIs

Key Competencies :

• Strong analytical and problem-solving skills.

• Good understanding of cloud data engineering concepts.

• Strong programming and SQL capabilities.

• Ability to work with large-scale data processing technologies.

• Good communication and client-facing skills.

• Strong troubleshooting and debugging capabilities.

Education:

• A Bachelors degree or equivalent qualification in Computer Science, Information Technology, Engineering, Data Engineering, or a related technical discipline is preferred.