Data Engineer at Anglo American

Job Description:
You will be responsible for:

• Develop ETL pipelines. The data transformations will be developed in Azure Databricks using Python and on Azure SQL using T-SQL and deployed using ARM templates
• Combine and curate data in a central data lake
• Serve data for application and analytics through a variety of technologies such as SQL, Server Synapse, Cosmos DB and TSI
• Build transformation pipelines into dimensions and facts and therefore a strong knowledge of standard BI concepts is mandatory
• Build stream pipelines leverage IoT Hub, Event Hub, Databricks streaming and other Azure stream technologies.
• Work in a fluid environment with changing requirements whilst maintaining absolute attention to detail.

This role is in the Business Support Services Department, Centurion at a Band 6 reporting to the Principal Data Analytics

Qualifications:

• IM Professional Qualification (Degree/Diploma) or Relevant Qualification

Technical Knowledge/Experience

• 5 years relevant experience in data engineering
• Strong experience of building large scale file shipping and pipelines, ideally using Azure services such as AzCopy and Azure Data Lake. Experience of managing unstructured file meta-data, conversions, standardisation and related workflows.
• Experience of building analysis jobs that scale on technologies such as Databricks or Azure Batch.
• Data Lake
• Data Modelling
• Data Architecture
• Azure Data Environment

Closing Date: 24th, March 2022