Data Engineering Lead at Intelligent Debt Management

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Certifications in Azure, Azure Databricks Data Engineering or similar or equivalent practical experience.

Candidates must be able to demonstrate:  

• Extensive proven experience as a data engineer or in a similar role, designing and implementing data architectures, data integration and ETL processes. Practical experience in Azure, Azure Data Factory and Azure Databricks is preferred.
• Proven experience in Python/PySpark and SQL/SparkSQL programming skills.
• At least 4 years of experience in Information Technology – specifically in data management.
• Strong knowledge of data management and storage solutions, including SQL Server Database, Azure Data Lake Storage or similar.
• Ability to lead and innovate in a fast-paced and evolving technology landscape.
• Strong communication and leadership abilities.
• Excellent attention to detail and problem solving skills related to data, data structures and algorithms.

Key responsibilities will include:

• Lead the design and implementation of scalable data solutions on the Microsoft Azure cloud platform, including Azure Databricks.
• Develop, maintain and optimise robust data pipelines for data ingestion, processing, and transformation to support analytics, machine learning and other data-driven services.
• Ensure data security, quality and integrity across all of the pipelines and related data processes.
• Define the mechanism through which skilled decentralized analytics teams can contribute to the engineering processes.
• Work closely with the Head of Data and enterprise architecture to ensure that the data engineering vision is aligned with the systems architecture and the data strategy.
• Collaborate with the data product owner, analysts, data scientists, and business stakeholders to understand data requirements and implement effective data workflows.
• Assist in data initiatives and support technical teams including architects, software developers, and data scientists.
• Provide leadership and mentoring to a small, skilled team of data engineers.