AWS Data Engineer
AWS Data Engineer with 4-6 years of experience to design and implement ETL pipelines and data processing applications using Python, PySpark, and SQL within the BFSI domain. The role involves building scalable data pipelines, optimizing Spark jobs, and working with cloud technologies like AWS.
- Compensation
- Not disclosed
- Location
- Hyderabad, India
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- 4+ years
- Category
- Data engineering
- Posted
Skills
Must have
- 4 - 6 years of experience
- BTech/MTech/MCA qualification
- Deep expertise in Banking, Financial Services, and Insurance (BFSI) systems
- Familiarity with Data warehousing concepts (ODS/Marts)
- Experience in end-to-end data processing pipeline design
- Strong programming knowledge in Python (OOP, exception handling, data structures)
- Hands-on experience with PySpark DataFrames and Spark SQL
- Understanding of Apache Spark architecture and distributed computing
- Strong SQL skills (joins, subqueries, CTEs, window functions)
- Experience developing ETL/ELT pipelines
- Knowledge of data formats like Parquet, JSON, and CSV
- Experience with Git and software development best practices
Nice to have
- Experience with cloud platforms (AWS, Azure, or Google Cloud)
- Familiarity with Databricks, Azure Data Factory, or AWS Glue
- Knowledge of data warehouses and lake house architectures
- Experience with Airflow or other workflow orchestration tools
- Familiarity with CI/CD pipelines, Docker, and automated testing
- Knowledge of Delta Lake, Kafka, or streaming data processing
Full description
**Job Title: Data Engineer
Experience Level: 4 - 6 years
Location: Hyderabad / PuneQualification: BTech/MTech/MCA
Mode of Work: Hybrid
We are seeking an experienced AWS Data
Engineer with deep expertise in Banking, Financial Services, and Insurance systems. The candidate
must be familiar at least in one Data ware housing concepts
such as ODS/Marts and must have done at least one end-to-end
data processing pipeline designing.
He/she should be capable of handling the ETL design and
implementation solely and would work towards creating a
positive and innovation friendly environment. The incumbent
would be responsible for creating new capabilities.
Responsibilities
- Develop, maintain, and optimize data processing applications using Python and PySpark.
- Build scalable ETL/ELT pipelines to process large volumes of structured and unstructured data.
- Perform data extraction, cleansing, transformation, aggregation, and validation.
- Work with Apache Spark for distributed data processing and large-scale data transformations.
- Write complex SQL queries and optimize database operations.
- Develop reusable Python modules, functions, and libraries.
- Optimize Spark jobs by analyzing execution plans, partitioning, caching, and performance bottlenecks.
- Work with file formats such as CSV, JSON, Parquet.
- Integrate data from multiple sources, including databases, APIs, and cloud storage.
- Implement error handling, logging, monitoring, and data quality checks.
- Collaborate with data engineers, data architects, analysts, and business stakeholders.
- Follow coding standards, version control practices, and Agile development methodologies.
Requirements
Requirements
- Strong programming knowledge in Python, including functions, OOP, exception handling, and data structures.
- Hands-on experience with PySpark DataFrames, Spark SQL, transformations, and actions.
- Good understanding of Apache Spark architecture and distributed computing.
- Strong SQL skills, including joins, sub queries, CTEs, and window functions.
- Experience developing ETL/ELT pipelines and handling large datasets.
- Knowledge of data processing formats such as Parquet, JSON, and CSV.