Data Engineer at Discovery Limited

Key Purpose

The Data Engineer is responsible to design and implement data pipelines and systems to enable clean and usable data to inform BI, MIS, data science and analytics.

Areas of responsibility may include but not limited to

• Engage with the macro-environment to look for new opportunities, capabilities and trends that would add value to the required data engineering work. This includes ensuring that data systems meet the company’s regulatory commitments and that data is reliable and efficient
• Adopt industry leading systems and ways of working to support best-in-class business capabilities for high performance database solutioning
• Actively participate in team, cross-discipline and vendor-driven collaboration sessions or forums to increase understanding of the working environment. This includes contribution and participation in the relevant Data Governance Frameworks
• Partner with Technology, Business and other stakeholders to ensure that database structures, programmes and applications meet business delivery requirements.
• Design and implement scalable end-to-end database solutions including:
• address issues of data migration i.e. validation, clean-up and mapping and consistently apply data dictionaries
• component design and development
• identify and resolve production and application development constraints
• develop prototypes and proof of concepts
• integrate new technologies and software into the existing landscape
• develop data set processes for data modelling, mining and production
• optimal utilization of big data
• documentation management and disaster recovery
• Monitor adherence to processes which support the prescribed data architectural frameworks and ensure development/delivery teams align to the required standards and methodologies
• Recommend appropriate interventions in own area of ways to improve data architectural compliance. This includes representation at relevant forums e.g. Architecture Board, Cloud Steerco
• Apply project management methodology to effectively engage various data initiatives. This includes collaborating with the data architect to produce data architectural designs for group wide projects and identifying project risks.
• In depth knowledge and understanding of:
• Data governance and data management frameworks
• BI and Warehouse design
• Metadata Management
• Service Oriented Architecture
• Data Integration: Extract Transform Load (ETL), batch architecture and frameworks. Skills required here include amongst others:
• Create and manage sophisticated T-SQL, ETL jobs, and solutions to load normalized table structures as well as Dimensional (SCD2) and Fact tables.
• Develop and maintain database stored procedures, views, and functions that support reporting data models.
• Convert ETL specs into ETL code which meets performance requirements of the ETL batch execution.
• Diagnose and resolve any ETL issues relating to ETL failure, incorrect outputs, and performance.
• Design and deploy table structures in alignment to reporting and data access/query needs.
• Develop, implement, and maintain change control of the ETL environment.
• Conduct unit testing and peer reviews of ETL code.
• Interact with Data Architect, System Architects and BI Leads to discuss solutions, challenges, and enhancements.
• Lead the creation, enhancement and implementation of newer Data Analytics and BI standards and processes.
• Create and maintain technical documentation of processes.
• Event Driven Architecture
• XML and related technologies
• SOA & Web Services
• Logical Database Design
• Statistical analysis and modelling advantageous
• Database architectures
• Hadoop-based technologies (e.g. MapReduce, Hive and Pig, Spark)
• SQL-based technologies (e.g. Oracle and MSSQL)
• NoSQL technologies (e.g. Cassandra and MongoDB)
• Data modelling tools (e.g. ERWin, Enterprise Architect and Visio)
• Python, C/C++ Java, Perl, R
• Predictive modelling, NLP and text analysis
• Machine learning
• Data mining
• UNIX, Linux, Solaris and MS Windows
• Cloud Architectures

Personal Attributes And Skills

• Excellent oral and written communication skills.
• Must be able to present technical solutions.
• Ability to coordinate and work cohesively with other Analysts and developers in the team (no ‘code-loners’).
• Solid experience in writing functional and technical specifications, process flows and source-to-target mappings.
• Experience creating databases from the ground up.
• In-depth understanding of data management (e.g., permissions, recovery, security, and monitoring)
• Strong SQL query writing ability (across database environments)
• Experience in ETL design and solid understanding of ETL solution architectures.
• Good understanding of ODS, data warehouse, data mart, cube, and database technologies.
• Strong analytical and problem-solving skills.
• Must be able to navigate ambiguity and complexity around business products and benefits and how they tie into data.
• Minimum intermediate experience with SSMS, SSIS, and T-SQL.
• Adaptability and comfort with fast-moving, changing environments.
• Innovation - ability to initiate new and better ways of doing things, using appropriate methods to identify opportunities, devise and implement solutions and measure impact
• Teamwork - capacity to work co-operatively with others to achieve shared goals
• Drive and Results Orientation - self-starter and originator who maintains high levels of activity and produces a consistently high quality output within agreed deadlines. Prompt and proactive in driving for success and sets demanding goals for self
• Assertiveness- able to hold one's own in the face of opposition and exert influence calmly, firmly and fairly
• Influencing and Impact (incl. Negotiation) - capacity to create appropriate levels of impact by using verbal and non-verbal communication skills to influence and/or negotiate win-win agreements to proposals, plans or ideas
• Verbal and Written Communication - capacity to use appropriate presentation and facilitation techniques to engage an audience and help them to understand, retain and respond appropriately to the message

Education And Experience

• Degree or Diploma in Computer Science / Informatics or than 7 years or more software development related experience
• Minimum 4 years’ experience in BI Data Warehouse development with degree or diploma