BI Data Engineer - Bellville at Sanlam Group

Main responsibilities

The performance of the BI Data Engineer can be described and measured by:

• Define a structured approach to problem solving and delivery against it.
• Create role specific design standards, patterns and principles
• Load large, complex data sets to and make data available for other data engineers
• Identify, design, and implement internal process improvements: automating manual processes, optimising data delivery, re-designing models for greater scalability
• Use modern development and modelling techniques and tools to implement BI and data management solutions, including data quality, metadata and reference data
• Engage with a wide range of technical stakeholders including data scientists, data analysts, business analysts, other data engineers and solutions architects
• Support data stewards to establish and enforce guidelines for data collection, quality improvements, integration, and processes
• Mentor Junior Engineers
• Working with other data engineers and data modelers, you will design, implement, and manage ETL programmes, data transformations and data pipelines that realise source to target mappings.
• Design, implement, and manage complex data vaults, data transformations and the data pipeline; generally using the Wherescape tool
• Monitor and fine-tune data pipelines and data transformations using SQL, Spark, NiFi on the Cloudera Hadoop stack
• Monitor and fine-tune data vaults, data transformations and stored procedures on the Cloudera Hadoop stack
• Identify, design, and implement vault access layers to enable BI products to leverage the data within data vaults
• Design and monitor user or system access to the data warehouse to ensure data security and integrity

What will make you successful in this role?

Qualifications

• Bachelor’s degree in Computer Science, Statistics, Informatics, Information Systems, Engineering or another quantitative field / National Diploma in an Information Technology related discipline preferred

Experience and skills:

• 3- 8 years related experience
• Application and data engineering background with a solid background in SQL is required
• Data architecture design and delivery experience preferred
• Experience in three (3) or more of the following areas is required:
• Database technologies (e.g. SAP Hana, Teradata or similar) and database development (Views, Functions and Stored Procedure development)
• Hadoop components including HDFS, Hive, Spark, Oozie and Impala
• Object-oriented/object functional scripting languages: Python, Java, Scala or related.
• ETL tools (e.g. SAP Data Services)
• Event/Streaming based data pipelines (e.g. Kafka or Nifi)
• Data warehousing (Kimball and Data Vault patterns are preferred) and dimensional data modelling (OLAP and MDX experience)