Data Engineer

Big Data Centre of Excellence is building state-of-the-art Data Analytics and Management systems used across various Jio Platforms. Primarily, we enable different businesses to ingest, process, manage and derive insights from data. We developed numerous data intelligence capabilities on top of these systems, which communicate with each other and help businesses make informed decisions. Big Data COE encourages the usage of open-source tools/services to build robust and resilient systems that form the core for various AI/ML capabilities.

This role is for a Sr Data Engineer with solid development experience who will focus on creating robust data pipelines and enhancing existing ETL processes. You will be an integral part of the development team, examining requirements and designing optimal solutions. The role is for a self-motivated individual with knowledge of the Hadoop ecosystem and various Hadoop tools/services. The candidate will perform hands-on activities including design, documentation, development and testing of new functionality.

Requirements:

• Basic Qualifications: 4+ years of relevant work experience with a bachelor's degree or master's degree, or a PhD.
• Excellent coding skills in Java/Scala/python especially in OOP constructs and concurrency systems; experienced in building highly optimal software systems.
• Well versed in shell scripting for working on Unix/Linux based systems.
• Experience in designing, building, tuning and troubleshooting distributed, scalable data pipelines and data streaming solutions.
• In-depth knowledge and hands-on experience of.
• Hadoop-based computing solutions including but not limited to: MapReduce, Spark, Hive, Tez, YARN, HBase, Kudu, Druid, Presto, Impala.
• Building real-time streaming solutions including but not limited to: Kafka, Spark Streaming, Structured Streaming, Apache Flink, Apache Beam, NiFi.
• Distributed search systems like Solr, Elasticsearch.
• Distributed memory grids like Ignite, Redis, Hazelcast.