Python Spark Developer

Job Description :

We are seeking a highly skilled Python Spark Developer with 8 to 12 years of experience in building and optimizing large-scale data processing applications.

The ideal candidate will possess strong expertise in Python, Apache Spark, Hadoop ecosystem technologies, and data engineering best practices.

This role involves developing high-performance data solutions, leading junior developers, and delivering scalable, low-latency applications capable of handling large volumes of data.

Key Responsibilities :

• Design, develop, and maintain scalable data processing applications using Python and Apache Spark.

• Build and optimize batch and real-time data processing pipelines using Spark Core, Spark SQL, and Spark Streaming.

• Develop high-performance, low-latency, and highly available data applications.

• Work closely with Hadoop ecosystem components to support large-scale data processing requirements.

• Perform coding, testing, debugging, troubleshooting, and deployment activities throughout the software development lifecycle.

• Optimize application performance through tuning, scalability improvements, automation, and resource utilization enhancements.

• Analyze and resolve production issues, ensuring system reliability and operational excellence.

• Collaborate with data engineers, architects, business stakeholders, and cross-functional teams to deliver data-driven solutions.

• Conduct code reviews and enforce development best practices and coding standards.

• Mentor and guide junior Python developers, providing technical leadership and support.

• Participate in design discussions, estimations, sprint planning, and Agile delivery activities.

• Maintain technical documentation and ensure adherence to organizational development standards.

Required Skills & Experience :

• 8 to 12 years of experience in Python development and data engineering.

• Strong proficiency in Python programming.

• Extensive hands-on experience with Apache Spark.

• Experience implementing Spark Core, Spark SQL, and Spark Streaming solutions.

• Strong experience with Pandas for data manipulation, transformation, and analysis.

• Experience working with Hadoop ecosystem technologies and distributed data processing environments.

• Strong understanding of data structures, algorithms, and software engineering principles.

• Experience designing and developing high-performance, scalable, and fault-tolerant applications.

• Expertise in performance tuning, optimization, load balancing, and automation.

• Strong debugging, troubleshooting, and problem-solving skills.

• Experience leading technical teams and mentoring junior developers.

• Familiarity with Agile/Scrum development methodologies.

• Strong communication, collaboration, and stakeholder management skills.

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.