Data Engineer - Template at Unitrans

Job Advert Summary    
As a Data Engineer you will be responsible for both on-premises and cloud-based data infrastructure, with a focus on migrating data pipelines to Azure Synapse Analytics and optimizing consumption side data provisioning. You will work closely with external consulting teams, in both data pipeline creation & maintenance and data management; and the analytics team, to understand analytics issues and requirements and translate them into scalable data solutions.

You will be working with internal company data as well as data provided by third party suppliers and partners. You would be involved in building and maintaining a cloud data lake, including the full process of ingestion, maintenance and provisioning of data from both internal and external data providers. Additionally, you will play a role in migrating business logic from analytics and reporting tools to Azure Synapse Analytics.

Minimum Requirements    
Experience and Qualification

• Bachelor's degree in Computer Science, Information Systems, or a related field OR proven capability in working with enterprise data systems.
• Azure Data Engineer Associate (DP-203) preferred
• Experience with Azure cloud services, particularly Azure Synapse Analytics.
• Strong understanding of cloud data solutions and architectures.
• Valid Driver's Licence with own transport 

Skills and Attributes

• Highly SQL proficient
• Python advantageous
• Process design and implementation
• Familiarity with APIs
• Familiarity with on-premises SQL Server databases and familiarity with migration processes to cloud platforms.
• Familiarity with Qlik Sense or similar reporting and analytics tools.
• Familiarity with enterprise systems and working with multiple databases.
• Knowledge of Microsoft Purview or similar data governance tools.
• Strong problem-solving skills and a demonstrated ability to learn new technologies independently.
• Proactive approach to problem solving
• Ability to function within various teams and environments, but also work independently
• Excellent communication skills

Duties & Responsibilities    

• Monitor, maintain and improve data pipelines on-premises and in Azure Synapse Analytics.
• Designing, developing, and testing large data pipelines to ingest, aggregate, clean, and distribute data models ready for analysis.
• Work with third party consultants to implement data lake architecture.     
• Build and maintain data pipelines to facilitate the smooth flow of data between various systems and databases.
• Provision datasets for consumption side users, focusing on consistency and data integrity.
• Migrate business logic from Qlik Sense to Azure Synapse Analytics.
• Migrate on premises data systems to the Azure Cloud.
• Test and maintain new and existing API connections.
• Contribute to and expand a data taxonomy to ensure consistency and clarity in data organization.
• Implement data security protocols across the entire data environment.
• Focused improvement initiatives of existing processes, systems, data, reports.
• Focused improvement initiatives of cross-functional interactions within the organisation.
• Willing to travel locallly and Internationally.