PySpark/Big Data Developer

BE | BTech | MCA | OR equivalent degree in Computer Science, IT or a related field candidates having 5 to 10 years of experience in Python, PySpark, SQL Unix + Big Data (Hadoop + Hive), Airflow, Datawarehousing, Data modelling.Required skills :

• Strong hands-on experience in Python Development.

• Extensive experience with PySpark / Apache Spark.

• Expertise in Data Engineering, SQL Unix + Big Data (Hadoop + Hive), Airflow, Datawarehousing, Data modelling and data processing frameworks.

• Strong knowledge of SQL and database concepts.

• Experience working with large-scale datasets in distributed environments.

• Understanding of Data Warehousing concepts and dimensional data modeling.

• Exposure to cloud platforms such as Azure, AWS, or GCP.

• Experience with version control tools such as Git.

• Experience with Kubernetes, Java or python + Spark SQL, Delta Lake.

• Knowledge of Airflow, ADF, or other orchestration tools.

• Experience in CI/CD implementation.

• Familiarity with SAP BW/4HANA, SAP S/4HANA, or SAP Banking solutions.

SAP Skills :

• Exposure to SAP FSDM/FPSL.

• Exposure to SAP Financial Services Data Management (FSDM).

• Understanding of SAP FSPL (Financial Services Package for Liquidity) + Big Data (Hadoop + Hive) processes and data structures.

• Knowledge of financial services, banking, risk, liquidity, or regulatory reporting domains is preferred.

Role & Responsibilities :

• Partner closely with Product Managers, Data Scientists, and AdOps teams in an Agile/Scrum environment to translate ambiguous business requirements into scalable, production-grade ETL/ELT data pipelines and dimensional data models.

• Design and maintain end-to-end cloud Lakehouse architectures on Python using PySpark, Databricks, and Delta Lake, establishing standardized frameworks for automated data ingestion, schema enforcement, and data reconciliation.

• Manage production workflow orchestration using Apache Airflow, building automated failure recovery, log monitoring, and SLA-driven alerting to guarantee 99.9% data pipeline uptime and reliable downstream analytics delivery.

• Mentored junior data engineers through pair programming and code reviews; established team-wide engineering best practices around PySpark performance tuning, Git version control, and CI/CD pipelines.

• Conducted routine cluster cost optimizations and query tuning across cloud infrastructure, adjusting Spark driver/executor configurations, dynamic allocation, and storage layouts to maximize compute efficiency.