Data Engineer IND

Job Purpose

The Data Engineer will be responsible for translating business requirements into Azure Cloud-based big data solutions for the data modernisation program. You will be responsible for building complex data pipelines, data warehouses, and data lakes using Azure services like Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.
The Azure DE will work closely with other members of the data Engineering team and product owners ensuring Data strategy and platforms fulfil current needs and are aligned with the overall vision of the organisation. They will be responsible for delivery of enterprise scale data transformation/modernisation programmes.
The ideal candidate will be an experienced data technologist and have a blend of experience across delivery and the practical application of contemporary data technologies. Experience in working within the Banking Industry (in banking, vendors or a large tech firm) is highly desirable.

Core Responsibilities

Develop high quality big data technologies-based solutions on Azure Cloud leveraging Azure Databricks and other services for advanced data processing, analytics tasks, optimizing performance and scalability.
Analyse current business practices, processes, and procedures as well as identifying future business opportunities for leveraging Microsoft Azure Data & Analytics Services
Develop best practices including reusable code, libraries, patterns, and consumable frameworks for Azure cloud-based data warehousing and data pipelines.
Work closely with the Azure Data engineering Manager and technical leads to ensure the execution of plans corresponds with what is being promised.
Data processing efficiency improvement, migrations from on-prem to public cloud platforms.

Experience Requirements

Overall 4-5 years of IT experience preferably in Data and Analytics implementation.
Minimum 3 years of Data engineering (Azure) delivery experience including analysis / development / configuration/ deployment.
2 years of implementation experience in Azure Data engineering projects
Experience in an MI / BI / Analytics environment (Kimball, lake house, data lake) is essential
Proficiency in Python/SQL/SparkSQL/PySpark
Expert level understanding on Azure Data Factory, Azure Synapse Analytics, Azure SQL, Azure Data Lake, Datra bricks, perview and Azure App Service is required.
Significant technical skills such as Transact SQL language, relational database skills are essential
Understanding of data governance and security best practices
Experience in Streaming and batch architecture (e.g., Kafka, Kafka streams, Spark, Flink)
Exposure in at least one distributed data processing framework: Spark (Core, Streaming, SQL), Storm or Flink, etc.
Should have worked on any NoSQL solutions like Mongo DB, Cassandra, HBase, Cosmos DB etc, or Cloud-based NoSQL offerings like DynamoDB, Big Table, etc.
Good Exposure in development with CI / CD pipelines. Knowledge of containerization, orchestration, and Kubernetes engine would be an added advantage.
Development of data pipelines using API ingestion and Streaming ingestion methods.
Experience in working in a fast-paced agile environment.