Data Engineering Lead at Progressive Edge

Cape Town

• A large established enterprise is looking to employ a Technical Lead : Data Engineering with deep technical knowledge the ability to lead and mentor a team of Data engineers.  You will require substantial expertise in a broad range of software development & programming fields.  Your knowledge of data analysis, end user requirements and business requirements analysis will help you develop a clear understanding of the business need and incorporate these needs into technical solutions.  You will also need a solid understanding of physical database design and the SDLC and have the ability to work well with and lead a team.

Responsibilities:

• Architects Data analytics framework.
• Translates complex functional and technical requirements into detailed architecture, design, and high performing software.
• Leads Data and batch/real-time analytical solutions leveraging transformational technologies.
• Works on multiple projects as a technical lead driving user story analysis and elaboration, design and development of software applications, testing, and builds automation tools.
• Development and Operations
• Database Development and Operations
• Policies, Standards and Procedures
• Communications
• Business Continuity & Disaster Recovery
• Research and Evaluation
• Coaching/ Mentoring

Qualifications:

• Essential: 4 years Bachelors degree in computer science, computer engineering, or equivalent work experience AWS Certification at least to associate level

Experience:

Essential:

• 5+ years Data engineering or software engineering
• 3-5 years demonstrated experience leading teams of engineers
• 2+ years Big Data experience
• 5+ years experience with Extract Transform and Load (ETL) processes
• 2+ years Cloud AWS experience
• At least 2 years demonstrated experience with agile or other rapid application development methods - Agile exposure, Kanban or Scrum
• 5 years demonstrated experience with object oriented design, coding and testing patterns as well as experience in engineering (commercial or open source) software platforms and large scale data infrastructures.

Knowledge & Skills:

Essential:

• Creating data feeds from on-premise to AWS Cloud (2 years)
• Support data feeds in production on break fix basis (2 years)
• Creating data marts using Talend or similar ETL development tool (4 years)
• Manipulating data using python and pyspark (2 years)
• Processing data using the Hadoop paradigm particularly using EMR, AWSs distribution of Hadoop (2 years)
• DevOps for Big Data and Business Intelligence including automated testing and deployment (2 years)
• Extensive knowledge in different programming or scripting languages
• Expert knowledge of data modeling and understanding of different data structures and their benefits and limitations under particular use cases.

 Further technical skills required:

• Capability to architect highly scalable distributed systems, using different open source tools.
• Big Data batch and streaming tools
• Talend (1 year)
• AWS: EMR, EC2, S3 (1 year)
• Python (1 year)
• PySpark or Spark (1 year) - Desirable
• Business Intelligence Data modelling (3 years)
• SQL (3 years)