Data Engineer (AWS) at Business Science Corporation

Job Overview

• You will work within a team solving complex problems to deliver real business value across a wide range of industries such as mining, telco’s, retail and financial services.
• You will help develop best practices and drive improvements in data engineering across the business.

Responsibilities, Activities and Key Delivarables

• Analyse and organise raw data
• Design data engineering solutions to meet business requirements
• Build scalable data pipelines that clean, transform and aggregate data from different sources using appropriate tools and technologies
• Collaborate with data scientists to prepare data sets for analytical modelling
• Identify ways to enhance data quality and reliability
• Manage technical delivery of projects
• Mentor junior data engineers

Job Specific Qualifications and Experience

Qualifications

• Graduate or Postgraduate ​degree in Computer Science, Information Technology, Statistics, Engineering or related technical field.

Requirements

• ​2-6 years of relational SQL database
• Solid understanding of Apache Spark as a distributed data processing framework
• Experience in designing, implementing and performance tuning solutions in Apache Spark
• Experience in complex data warehouse and data lake design
• Experience in building, optimising and managing data pipelines for large and complex datasets
• Experience using AWS cloud data platform technologies:
• Amazon S3
• Amazon Redshift
• AWS Glue
• Amazon Athena
• Amazon Elastic MapReduce (EMR)
• AWS Step Functions
• AWS Lambda to implement data pipelines and analytical solutions
• Experience in Python would be beneficial
• Experience with streaming technologies such as Spark streaming or Kafka would be beneficial
• A relevant data engineering certification would be beneficial

Behavioral Competencies

• Problem Solver: Can identify problems and opportunities and use critical thinking skills to develop innovative solutions in response.
• Client Focused: Seeks to understand clients and establish effective collaborative working relationships.
• Proactive: Communicates proactively with project stakeholders and takes action to achieve goals.
• Resilient: Copes effectively under pressure or opposition (time or ambiguity).
• Team Orientated: Enjoys working in a team and has a collaborative and inclusive leadership style.
• Quality Focused: Focused on delivering high quality solutions.
• Communication Skills: Strong verbal and written communication skills. Can effectively present complex technical concepts to both technical and non-technical audiences.