Indian Institute of Science
Data Engineer - Python/ETL
- Compensation
- Not disclosed
- Location
- Not listed
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- Not specified
- Category
- Data engineering
- Posted
Skills
Must have
- - Solid understanding of available data sources and APIs in city solutions, including the ability to evaluate data quality, availability frequency, and common issues like data stream repetitions, as w
- - Experience in developing ETL modules in Python for data ingestion into data exchange using REST APIs and streaming protocols such as AMQP and MQTT, including containerizing adapters and creating dat
- - Good understanding and expertise in working with geospatial data (vector and raster), with experience in processing geospatial data using GIS tools like QGIS and ESRI, and skilled in analysing, tran
- - Proficient in Linux, with experience in GIT version control and cloud computing platforms.
- - Strong understanding of IoT, GIS, Big Data, and Cloud applications aimed at improving operational efficiency and service delivery, with a commitment to creating a positive societal impact.
- - Experience in containerization using Docker, with familiarity in Kubernetes for orchestration is a plus.
- - Knowledge of monitoring and logging tools such as Prometheus, Grafana, and Logstash would be a strong plus.
Full description
Scope of Work :
- Collect requirements and collaborate with city officials, agencies, system integrators, solution providers, and other data sources to integrate relevant datasets into the data exchange platform.
- Deploy and manage data exchange, while supporting application providers in building solutions for optimised management and citizen services.
Key Responsibilities :
- Evaluate APIs and datasets, create data models, develop software ETL modules, perform unit testing, and deploy them in cloud environments.
- Develop ETL modules in Python to ingest data into data exchange using REST APIs and streaming protocols such as AMQP and MQTT.
- This includes containerizing the adapters, creating data models, and catalogue entries according to data exchange specifications.
- Follow best practices for software development and adhere to Agile methodology throughout.
Required Skills :
- Excellent software design and programming skills in Python, and Java/JavaScript, with proficiency in data model technologies such as JSON-schema, JSON-LD, and Protobuf, along with strong functional testing skills.
- Solid understanding of available data sources and APIs in city solutions, including the ability to evaluate data quality, availability frequency, and common issues like data stream repetitions, as well as familiarity with various data fields and their meanings, and data structure formats (JSON, GeoJSON).
- Experience in developing ETL modules in Python for data ingestion into data exchange using REST APIs and streaming protocols such as AMQP and MQTT, including containerizing adapters and creating data models and catalogue entries according to data exchange specifications, while adhering to best practices in software development and Agile methodology.
- Good understanding and expertise in working with geospatial data (vector and raster), with experience in processing geospatial data using GIS tools like QGIS and ESRI, and skilled in analysing, transforming, and visualising geospatial data.
- Proficient in Linux, with experience in GIT version control and cloud computing platforms.
- Strong understanding of IoT, GIS, Big Data, and Cloud applications aimed at improving operational efficiency and service delivery, with a commitment to creating a positive societal impact.
- Experience in containerization using Docker, with familiarity in Kubernetes for orchestration is a plus.
- Knowledge of monitoring and logging tools such as Prometheus, Grafana, and Logstash would be a strong plus.