Data Engineer at Woolworths

The main purpose of a data engineer is to source and prepare data for consumption by Analysts and Data Scientists in the relevant forms and frequency to enable growth and optimal operations management for the enterprise. This IT role requires a significant set of technical skills, including a deep knowledge of cloud database design and multiple programming languages.

KEY RESPONSIBILITIES:

Data Management/ Analysis

• Create and maintain optimal data pipeline architecture,
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using AWS ‘big data’ technologies.
• Implement analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.  Analyzing raw data, as well as develop and maintain datasets
• Prepare data for prescriptive and predictive modelling in both batch and streaming formats

Maintain and support existing systems

• Improve data quality and efficiency
• Identify opportunities for data acquisition
• Manage incidents related to cloud data environment

Effective operations

• Ensure data separation and security across national boundaries through multiple data centres and AWS regions.
• Implement data tools for analytics team members that assist them in building and optimizing our products
• Develop and test architectures that enable data extraction and transformation for predictive or prescriptive modelling
• Establish appropriate data monitoring processes to ensure availability and service levels are maintained                 

Relationship Building

• Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Work with data and analytics experts to strive for greater functionality in our data systems

KEY COMPETENCIES

• Strong analytical skills and the ability to combine data from different sources.
• Cloud Data Lake and Database development and maintenance
• Detail-oriented, with excellent organizational skills
• Strong communication and problem-solving skills
• Business and commercial acumen

MINIMUM QUALIFICATION

• Appropriate degree/ diploma in Computer Science/Information Technology/Math
• Appropriate certification in cloud data engineering advantageous

EXPERIENCE REQUIRED

• Minimum of 5 years’ experience in cloud data engineering and design
• Strong experience in working with AWS cloud services: EC2, EMR, RDS, Redshift, S3 working on Terraform and object-oriented/object function scripting languages: SQL and Python
• Experience building and optimizing ‘big data’ data pipelines, architectures and data sets.
• Strong analytic skills related to preparing both structured & unstructured datasets from streaming and batch processes.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• A successful history of manipulating, processing and extracting value from large disconnected datasets.