Lead Data Engineer: AWS at Absa South Africa

Job Summary

• Leveraging a thorough understanding of the business data requirements & own data subject matter expertise (ahead of trend), set data architecture direction & lead the agile solution design, deployment, implementation & ongoing optimization of enterprise wide data retrieval, storage and distribution across an estate.

Job Description

• Providing consultancy and hands-on help to development teams designing, developing and releasing solutions to Cloud based infrastructure
• Provide hands-on support resolving issues with Cloud based infrastructure
• Advise on best practice when designing and implementing Cloud based solutions
• Architectural guidance on next iteration of technology stack.  AWS Set Up: Account, Configuration and getting the data into the Production instance of AWS
• Application Development including data pipeline, data storage, data analytics as well as front end and back end of technology stack
• Data engineering and data lineage (data life cycle) considerations on how to manage the pipeline from source to analytics engine
• Collaborate with data science teams & broader business & technical stakeholders to analyse customer data requirements & leverage analysis to identify & accurately spec / frame & set the appropriate data architecture for the business area
• Take accountability for deeply understanding the technical landscape and bank wide architecture that is connected to or dependent on the business area supported in order to effectively design & deliver data solutions (architecture, pipeline etc.)
• Leverage expertise in analytical & creative problem solving to synthesise data solution designs (build a solution from its components) beyond the analysis of the problem
• Lead design thinking processes to successfully deliver data solution blueprints & associated roadmaps
• Lead the end to end solution design & implementation process for all data retrieval, storage & distribution solutions including contributing to and in some cases lead all phases of the development lifecycle including e.g. design process
• Build analytics tools that utilize the data pipeline by quickly producing well-organised, optimized, and documented source code & algorithms to deliver technical data solutions
• Create & Maintain Sophisticated CI / CD Pipelines (authoring & supporting CI/CD pipelines in Jenkins or similar tools and deploy to multi-site environments – supporting and managing your applications all the way to production)
• Debug existing source code and polish feature sets.
• Assemble large, complex data sets that meet business requirements & manage the data pipeline
• Build infrastructure to automate extremely high volumes of data delivery and
• Creatively solve data volume and scaling challenges.
• Create data tools for analytics and data science teams that assist them in building and optimizing data sets for the benefit of the business
• Ensure designs & solutions support the technical organisation principles of self-service, repeatability, testability, scalability & resilience
• Apply general design patterns and paradigms to deliver technical solutions
• Inform & support the infrastructure build required for optimal extraction, transformation, and loading of data from a wide variety of data sources.
• Meaningfully contribute & ensure solutions align to the design & direction of the Group Architecture & in particular data standards, principles, preferences & practices. Short term deployment must align to strategic long term delivery.
• Meaningfully contribute & ensure solutions align to the design and direction of the Group Infrastructure standards and practices e.g. OLA’s, IAAS, PAAS, SAAS, Containerisation etc.
• Strategically & operationally monitor the performance of data products and services ensuring ongoing optimization & cost to value for our businesses (think bank wide)
• Stay ahead of the curve on data retrieval, storage & distribution technologies & processes (global best practices & trends) to ensure best practice

Pre-requisite Skills:

• Pro-level skills: 
• Python 2 & 3
• Nodejs
• Terraform
• C#
• AWS Certified Dev Associate
• AWS Certified Solutions Architect Associate
• TensorFlow
• PySpark
• AWS Data Analytics Specialty

Additional Certifications: 

• AWS Certified Sysops associate
• AWS Certified Solutions Architect Professional
• AWS Certified DevOps Professional

Key understanding of – additional benefits:

• Absa ServiceNow and the CMDB;
• Technology platforms, software
• Confluence, Bitbucket, JIRA & Quicksight