Data Engineer

Responsibilities :

• Design, Develop, Maintain and own data engineering projects built from requirements by different stakeholders, including but not limited to product teams, data scientists, data analysts and BI.

• Fulfill any requirement related to data pipelines, data warehouses and databases required by the application.

• Create ingestion pipelines to collect data from different sources and push to targets.

• Design and Create SQL, tables, procedures, functions etc.

• Operate and Maintain existing Hadoop Cluster.

• Write Spark Tasks/Applications mainly in Python.

• Evaluate, test and update data applications according to changing requirements.

• Monitor and report data systems health and security.

• Support internal teams to use our systems, troubleshoot & identify issues, and resolve them.

• Follow best coding practices and participate in the code review process.

• Use efficiently the development environment, compiling tools, version control and bug tracking systems for all projects.

**Technical Competencies :Must Have :

• 3 - 5years of experience in Data Engineering.**

• Good Knowledge/Experience in Python. Hadoop Stack, Oozie, Spark, Kafka & Airflow

• Hands-on in writing SQL queries, Stored Procedures and Database code

• Good Knowledge with concepts, design and mechanics of traditional databases, data warehouses and data lakes

• Good Knowledge with ETL, ELT, Transformation, Processing, Computation etc.

• Experience with relational databases such as MySQL, PostgreSQL, MariaDB.

• Good Knowledge/Experience with Git, CI/CD, and writing unit tests.

• Good Knowledge/Experience working on an agile development project.

• Good Knowledge/Experience with Jira and the Atlassian tool suite.

• Good Knowledge/Experience in Linux Shell Scripting

Nice to Have :

• Experience with NoSQL databases such as Aerospike, DynamoDB, Redis, Elasticsearch, Cassandra, MongoDB.

- Good to have experience in BigQuery.

• Knowledge on one of the ETL tools

- AWS Lambda, Kinesis, SQS and SNS

• Experience with performance and scalability issues.

• Non Technical Competencies

• Good English and communication skills.