Data Engineer at Discovery Limited

Key Purpose

• The Senior Data Engineer is responsible for the design, development, and optimisation of scalable, enterprise-grade data platforms on Azure, with Azure Synapse Analytics at the core.This role enables the delivery of high-quality, trusted, and reusable data assets to support analytics, reporting, and AI use cases. The incumbent will play a key role in advancing a modern data lakehouse architecture, ensuring data is governed, performant, and aligned to business value outcomes.

Areas of responsibility may include but are not limited to:

Design

• Translate business requirements into technical designs adhering to Discovery Insure processes, standards and guidelines, considering performance, security, scalability, and cost.
• Define and evolve medallion architecture patterns in Microsoft Fabric (Bronze/Silver/Gold), including ingestion, transformation, and curated consumption layers.
• Produce and maintain modern data models:
• OLTP-aligned models where needed
• Dimensional models (star/snowflake) for analytics
• Curated “Gold” layer data products (conformed dimensions, business-ready facts)
• Design ingestion and transformation strategies across Fabric components (Lakehouse/Warehouse, Data Pipelines, Dataflows Gen2, Notebooks/Spark).
• Establish data quality, lineage, and governance requirements (e.g., data contracts, validation rules, auditing, retention).
• Liaise with relevant parties where clarification of business requirements or resolution to technical issues is needed.
• Research and recommend effective solutions to technical issues that arise (e.g., performance bottlenecks, modelling trade-offs, orchestration patterns).
• Estimate development timelines based on business requirements and technical complexity; identify risks and dependencies early.

Development

• Construct robust, maintainable, scalable, optimally performing solution components in line with technical specifications, following prescribed process, standards and procedures.
• Develop and optimise:
• T-SQL for Fabric Warehouse/Azure SQL (as applicable)
• ELT/ETL transformations (SQL/Spark) aligned to the medallion pattern
• Reusable frameworks for ingestion, validation, and incremental processing
• Build and maintain orchestration using Fabric Data Pipelines (scheduling, dependencies, retries, parameterization).
• Implement patterns for incremental loads and change processing (e.g., CDC-like approaches, watermarking, SCD Type 1/2 where appropriate).
• Apply performance engineering:
• Partitioning strategies, incremental refresh patterns
• Query optimisation and model simplification
• Efficient file formats and table design for analytical workloads

Testing

• Conduct unit testing and fix any defects found, aiming for high-quality releases.
• Verify build stability and quality with development team before releasing to test team, aiming to release with zero defects.
• Build and maintain test data sets, validation rules, and reconciliation checks across Bronze/Silver/Gold.
• Assist the Business Analyst in ensuring the test pack includes relevant scenarios and test data.
• Consult and assist in reviewing risk / impact of defects found in testing and assist with fixing where necessary.

Knowledge and Skills:

• Modern Data Modelling & Analytics
• Modern data modelling (OLTP where needed, dimensional modelling, curated analytical models, semantic consistency)
• Strong understanding of medallion architecture concepts (Bronze/Silver/Gold, data quality gates, curated data products)
• Metric definition discipline (facts, dimensions, conformed entities, grain, business rules)
• Microsoft / Azure Data Platform
• Microsoft Fabric: Lakehouse/Warehouse concepts, Pipelines, Dataflows Gen2, Notebooks/Spark (as applicable)
• T-SQL expertise (complex querying, optimisation, robust patterns)
• Performance tuning and optimisation for analytical workloads (SQL/Spark and modelling considerations)
• Understanding of security patterns (RBAC, least privilege, workspace access, data access controls)
• Engineering Excellence
• Strong SDLC practices (requirements → design → build → test → release)
• Source control and peer review culture (Git; branching strategies; PR discipline)
• Data quality and validation frameworks (reconciliation, anomaly checks, rule-based validations)
• Documentation and maintainability: clear design artifacts, data dictionaries, lineage-ready documentation

Working knowledge of:

• Software development within SDLC.
• Unit Testing and quality practices.
• Data modelling and design of database structures.
• Practical understanding of cloud data solution concepts (security, orchestration, monitoring, cost-awareness).

Advantageous

• Power BI semantic modelling (star schemas, measures, performance considerations)
• Governance tooling awareness (cataloging, lineage, data classification)
• Exposure to CI/CD for data solutions (deployment pipelines, environment promotion concepts)

Education and Experience

Education:

• Matric (Essential)   
• BSc Computer Science or equivalent 3-year qualification - (advantageous)

Minimum Experience:             

• 5+ years consistent experience in Microsoft data development with demonstrable delivery of complex solutions (senior-level capability).
• Strong experience with SQL development and performance optimisation.
• Proven experience designing and implementing data models and data structures in support of operational and analytical requirements.
• Experience delivering data solutions aligned to layered architecture patterns (or demonstrable capability to adopt medallion practices quickly).

Advantageous

• Experience modernising or migrating data workloads from on-prem to cloud.
• Experience building curated “Gold” layer models for enterprise reporting/analytics.