Jobs Career Advice Post Job
X

Send this job to a friend

X

Did you notice an error or suspect this job is scam? Tell us.

    • Senior Data Engineer at Mediro Recruitment

    Posted: Nov 18, 2026
    Expires: Not specified
    Save
    Facebook
    X
    LinkedIn
    Email
    Copy link
  • Get new jobs in your inbox
    • @gmail.com
    • @yahoo.com
    • @outlook.com
  • We specialise in providing Information and Communication Technology products, services, and solutions to corporate and public sector clients. Our depth of expertise and passion for delivering value-adding solutions allows us to align our client’s critical applications with their evolving strategies and business goals.

    View All Jobs at Mediro Recruitment


     

    Senior Data Engineer

    About the role

    • Our client in Fourways is looking for a Senior Data Engineer to join their team. The Data Engineer will provide hands-on technical delivery and optimisation support for the Digital Agriculture Agri Data Hub, developed on Azure Databricks. The platform integrates structured data from multiple on-premises databases and unstructured SharePoint content to support business intelligence, agronomic analytics and AI-driven capabilities. The project has progressed into User Acceptance Testing (UAT), and the resource will focus on resolving defects, stabilising data pipelines, optimising performance and supporting production readiness.
    • The role requires a technically strong, delivery-focused engineer who can operate independently within an established architecture and contribute to ongoing MVP improvements.
    • The Data Engineer will join an existing Digital Agriculture implementation focused on delivering a centralised Agri Data Hub within Azure Databricks. The solution incorporates structured data ingestion from on-premises databases into a Medallion Architecture, curated Gold-layer data products supporting BI reporting, and a Generative AI/LLM knowledge agent (Agripedia) consuming unstructured SharePoint information.
    • Current project position: Analysis and design, core architecture, Azure networking, CI/CD pipelines, Gold-layer data modelling and initial BI and LLM development have been completed. The BI reports and Agripedia solution are currently in UAT, with defect resolution and production readiness activities underway.
    • Technical delivery and stabilisation: Review and understand the existing Databricks implementation, ingestion pipelines, data transformations and Gold-layer models. Investigate and resolve defects identified during UAT, troubleshoot integration and data-quality issues, and implement improvements required to stabilise the solution.
    • Pipeline development and optimisation: Develop, maintain and optimise ingestion pipelines for structured data from on-premises databases. Implement scalable transformation logic using Spark, PySpark and SQL, ensuring that pipelines are performant, secure, cost-efficient and supportable. Apply Delta Lake optimisation techniques and maintain consistency across the established Medallion Architecture.
    • Production readiness and deployment: Support the transition from UAT into production, including validation of data accuracy, pipeline execution, error handling, monitoring and recovery processes. Work within existing Azure DevOps and CI/CD deployment processes, ensuring changes are appropriately tested and deployed.
    • Cost, performance and operational improvements: Monitor and optimise Databricks compute usage, pipeline performance and associated cloud costs as the solution moves into production. Identify opportunities to improve operational efficiency, stability and maintainability.
    • AI and unstructured data integration: Support the existing Agripedia LLM knowledge-agent solution where data engineering input is required, particularly relating to SharePoint data ingestion, integration, data preparation and ongoing optimisation. Previous experience supporting AI/LLM data pipelines would be advantageous.
    • Governance, documentation and knowledge transfer: Adhere to existing enterprise governance and architecture standards, contribute to data-quality controls, and update technical documentation where required. Collaborate with the Data Architect and internal technical teams to support knowledge transfer, troubleshooting and further enhancements.
    • Ongoing MVP enhancements: As the solution is being delivered iteratively, support further development and refinement of data pipelines, curated data products and platform capabilities based on business priorities, testing feedback and production requirements.

    What they're looking for

    • Databricks, Microsoft Azure, Spark, PySpark, SQL, Delta Lake, Medallion Architecture (Bronze/Silver/Gold), Azure DevOps, CI/CD, Data Integration, Data Quality, Performance and Cost Optimisation. Exposure to Generative AI/LLM solutions and SharePoint integration would be advantageous.
    • Strong intermediate to senior-level engineer with proven hands-on Databricks experience in production environments. Must demonstrate strong troubleshooting and problem-solving capabilities, with the ability to understand existing solutions, identify root causes and implement practical improvements. Comfortable working within an established architecture and governed enterprise environment, collaborating with architects, technical teams and business stakeholders. Must be proactive, adaptable and capable of taking ownership of technical activities with minimal supervision. Experience supporting UAT, production deployments and iterative MVP delivery is important.

    Check how your CV matches this job

    Method of Application

    Interested and qualified? Go to Mediro Recruitment on jobs.recruit360.co.za to apply

    Build your CV for free. Download in different templates.

  • Save Report job
  • Get new Data, Business Analysis and AI jobs on TelegramNew jobs like this, as they are posted
  • Send your application

Career Advice

View All Career Advice
 

Subscribe to Job Alert

 

Join our happy subscribers

 
 
Send your application through

GmailGmail YahoomailYahoomail