Founded in March 2006, the company started off offering 'deep skills’ technical support and consulting to other IT companies or to companies whose own IT departments were overwhelmed or who lacked the technical skills required for specific projects.
As our customer base grew we started taking on more and more SME clients who were struggling to find decent ...
As a Senior Data Consultant, you will play a critical role in designing, developing, and maintaining data pipelines to enable data-driven decision-making across our clients' businesses. You will work closely with a small but highly skilled team of data engineers, data scientists, and business analysts to deliver high-impact solutions. You'll be expected to bring both technical expertise and a collaborative mindset, helping to shape the architecture and development of data solutions on cloud platforms.
Key Responsibilities
Design, build, and maintain scalable data pipelines for the collection, transformation, and storage of data.
Work with large, complex datasets to ensure efficient processing and integration.
Develop data engineering solutions using Azure technologies and PySpark for distributed data processing.
Implement ETL processes and automate data workflows to support analytics, reporting, and business intelligence initiatives.
Collaborate with data scientists, business analysts, and stakeholders to understand data requirements and deliver actionable insights.
Ensure data quality, consistency, and reliability across all pipelines and datasets.
Optimize data models, storage, and processing workflows for performance and scalability.
Contribute to the architecture and design decisions for cloud-based data platforms.
Mentor junior team members and foster a collaborative and innovative team culture.
Troubleshoot and resolve issues related to data pipelines, ensuring high availability and performance.
Key Requirements
Essential:
8-10 years of experience in data engineering or a related role with a strong background in building and optimizing data pipelines.
Expertise in PySpark for large-scale data processing and distributed computing.
Strong experience with Azure data technologies (Azure Data Factory, Azure Databricks, Azure SQL Database, etc.).
Proficiency in SQL and experience working with relational and NoSQL databases.
Experience with cloud-based data platforms and services, especially in Azure.
Solid understanding of ETL processes and data modeling techniques.
Strong problem-solving and troubleshooting skills.
Experience working with version control systems (e.g., Git) and continuous integration/continuous deployment (CI/CD) practices.
Strong communication skills and the ability to work effectively within a collaborative, small-team environment.
A proactive, self-starter attitude with a passion for data engineering and technology.
Preferred
Experience with data warehousing solutions (e.g., Azure Synapse Analytics, Snowflake).
Familiarity with other big data technologies (e.g., Hadoop, Kafka, Spark Streaming).
Experience with infrastructure as code (IaC) tools such as Terraform or Azure Resource Manager (ARM).
Knowledge of containerization technologies (e.g., Docker, Kubernetes).
Johannesburg vs. Cape Town: How Job Opportunities CompareA comparison between job opportunities in Johannesburg and job opportunities in Cape Town job opportunities, including salaries, employment growth, industries hiring and career prospects.
20 Initiatives to Boost Employee EngagementAre you struggling with improving employee engagement at work? This article covers everything from better communication to building a strong workplace culture.
30 Common Interview Mistakes to AvoidThis piece examines 30 of the most common mistakes applicants make at interviews, so you know how to better avoid them.
Gmail
Yahoomail