Job Purpose:
- The primary purpose of this role is to take a leading role in designing, implementing, and optimising CIPC's data infrastructure to support strategic goals. This position is central to building scalable, high-performance data pipelines, driving data engineering best practices, and ensuring the delivery of robust data solutions that empower data scientists and analysts to generate critical business insights.
Required Minimum Education / Training
Candidates must meet one of the following requirements:
Formal Education Pathway (Preferred):
- Education:Bachelor’s Degree / Advanced Diploma in Computer Science, Engineering, or a related technical field. Advanced degrees are a plus.
- Added Advantage Certifications: Holding one or more of the following advanced professional certifications is a strong advantage:
- Specialized Tools: Databricks Certified Data Engineer (Professional or Associate), Confluent Certified Developer for Apache Kafka, or Certified Kubernetes Administrator (CKA).
- Cloud Platforms: Google Cloud Certified Professional Data Engineer, Microsoft Certified: Azure Data Engineer Associate, or AWS Certified Data Analytics – Specialty.
- Data Management: Certified Data Management Professional (CDMP) (Practitioner or Professional level).
Specialized Certification and Experience Pathway (Alternative):
- Education: Senior Certificate (NQF 4) and relevant technical certifications
Mandatory Certifications:The candidate must hold a combination of at least two advanced professional certifications, which must include one specialized tool certification and one cloud platform certification:
Specialized Tool Certification (MUST include one):
- Databricks Certified Data Engineer (Professional or Associate)
- Confluent Certified Developer for Apache Kafka
- Certified Kubernetes Administrator (CKA) or Certified Kubernetes Application Developer (CKAD)
Cloud Platform Certification (MUST include one):
- Google Cloud Certified Professional Data Engineer
- Microsoft Certified: Azure Data Engineer Associate
- AWS Certified Data Analytics – Specialty
Added Advantage Certification (Choose one more from any category):
- Certified Data Management Professional (CDMP) (Practitioner or Professional level)
- DAMA Certified Data Management Professional (CDMP)
- Any other certification listed above not yet counted.
Required Minimum Experience
- Experience: Minimum 5+ years (preferred 8+ years) of proven experience as a Data Engineer or in a similar technical role, with a strong track record of building scalable data solutions.
- Experience: Minimum 10+ years of proven experience as a Data Engineer or in a similar technical role, with a strong track record of building scalable data solutions.
Key performance areas
Data Pipeline & System Development
- Pipeline Design and Implementation: Lead the design, build, and maintenance of scalable and secure data pipelines (batch and real-time) and databases to process complex, high-volume structured and unstructured data.
- ETL/ELT Development: Develop and optimize ETL (Extract, Transform, Load) processes to efficiently transform raw data into usable, high-quality formats for analysis and consumption.
- Infrastructure Optimization: Manage and optimize data warehousing solutions (e.g., Databricks, Snowflake, Redshift, BigQuery, Synapse) and implement and maintain data storage solutions, including SQL and NoSQL databases.
- Automation: Automate data processing tasks using frameworks like Apache Airflow and optimize deployment and orchestration to improve efficiency and reduce manual effort.
Data Quality, Performance & Compliance
- Performance Monitoring: Monitor data pipeline performance, troubleshoot issues promptly, and optimize data processing frameworks to handle increasing data volumes with low latency.
- Quality and Standards: Develop and enforce standards and best practices for data quality, security, documentation, and compliance across all data systems and processes.
- Architecture Contribution: Ensure data pipelines and databases are optimized for performance, security, availability, and scalability, and contribute actively to overall data architecture decisions.
Collaboration, Mentorship & Strategy
- Stakeholder Collaboration: Work closely with data scientists, data analysts, and business teams to understand their data needs, ensure data accessibility, and deliver solutions tailored for their analysis requirements.
- Technical Leadership: Serve as a technical leader, coach, and mentor for junior data engineers and adjacent data and engineering teams.
- Project Leadership: Lead end-to-end data engineering projects, including requirements gathering, technical deliverable planning, output quality control, and stakeholder management.
- Strategy Contribution: Contribute technical expertise to the development and evolution of the CIPC data strategy.
Minimum Functional Requirements (Technical Skills & Knowledge)
- Core Programming: Expertise in programming languages such as Python, Java, or Scala.
- SQL Mastery: Advanced proficiency with SQL and deep experience in database optimization techniques for high-volume data.
- Big Data & Distributed Systems: Strong hands-on experience with distributed systems and big data technologies, including Apache Spark, Hadoop, or Flink.
- Cloud Data Platforms: Strong knowledge of cloud-based data platforms and their services across major providers (AWS, Azure, and GCP).
- ETL/Orchestration Tools: Proven experience with ETL tools and orchestration frameworks (e.g., Apache Kafka, Apache Airflow, Apache Spark).
- Architecture Design: Experience in designing and implementing data architectures that specifically support large-scale data processing and machine learning initiatives.
- Soft Skills: Strong problem-solving and critical thinking skills, excellent interpersonal skills, and the ability to work effectively with cross-functional teams.
- Mentorship: Proven experience leading and mentoring junior data engineers.
Gmail
Yahoomail