Cloud DataOps Engineer at Vodacom

Role Purpose

The Cloud DataOps Engineer role based within the Technology Business Unit.

• The Cloud DataOps Engineer is to be a cross-functional person responsible for preparing data ingestion pipelines, implementing monitoring and defining and implementing data quality mechanisms. This includes automating data monitoring, alerting, and checking across various stages of data transformation and projection. As well as ensuring effective data quality controls are inherent in the data ingestion patterns and data governance to serve quality and trusted data to our customers. Additionally, support and maintain production data pipelines.

Your responsibilities will include:

• Perform and prepare multiple data platforms to integrate data, be responsible for data quality control, and formulate data integrity solutions.
• Implement proactive monitoring, alerting, trend analysis, and robust applications.
• Implement and comply with the data governance policy and related controls across multiple data platforms. 
• Implement data reporting and execute data QC applications/systems.
• Contribute to enhancing and improving ETL processes and data ingestions across multiple platforms. 
• Work effectively on an Agile team and collaborate well with your other team members.
• Provide support to other teams if any optimization or the troubleshooting on the performance of the application to avoid errors.
   

The ideal candidate for this role will have:

• Matric / Grade 12 essential 
• 3 Year diploma / degree in Computer Science, Data Analytics, Data Science or related field essential 
• 3+ years of overall IT experience with Big Data, Advance Analytics, Data Warehousing and Business Intelligence
• Relevant Cloud experience
• Relevant cloud certification at professional or associate level would be advantageous. 
• Solid experience in building batch and stream data pipelines.
• Agile exposure, Kanban or Scrum

Core competencies and knowledge:

• In-depth knowledge of data as a product & Information best practices.
• Intermediate to advanced level experience in designing, building and managing data pipelines for batch and streaming applications
• Experience with performance tuning for batch based applications like Hadoop, including working knowledge using Nifi, Yarn, Hive, Airflow and Spark.
• Experience with performance tuning streaming based applications for real-time data processing using Kafka, Confluent Kafka, AWS Kenesis, GCP pub/sub or similar
• Working experience within the ML and Analytics lifecycle capabilities such as Data Pipelines, Data Processing, Data Storing, Model Lifecycle, Data Operations, Data Management & Data Governance.
• Experience in using a wide range for data tools such as Hadoop, Spark, Hive, Cassandra, Airflow, Kafka, Flink, AWS services, GCP Services, etc.
• Working experience with Cloud platforms such as OpenShift, AWS and GCP.
• Working experience with CI/CD.
• Java and Python programming