Team Lead: Data Engineering at Clickatell

Responsibilities of the Role

The Data Engineering Lead has a strong focus on programing and will drive and maturity our big data engineering capabilities within our products and organisation.

You will be responsible for the development of data products and advanced technical thought leadership for the organisation, and delivering fit-for-purpose data products for consumption by advanced analytics.

The Data Engineering Team Lead will manage the full life cycle of big data engineering, from design, implementation through to production.

Solution Design and Development

• Participate in grooming sessions with technical and business stakeholders to convert requirements into technical solutions that support the development of Clickatell’s products and platforms, for both real-time and batch environments.
• Innovate and collaborate with application developers, data architects, database administrators, data engineers and analytics developers to define and evolve our data model and select appropriate data technologies that fit the requirements.
• Champion the design and development best practices, ensuring high performance and scalability
• Perform tests and produce results to ensure accurate compliance with requirements.

Leadership

• Lead and manage a team of Data Engineers
• Manage and Approve team leave and on-call support schedules
• Plan and assign work for the team to deliver on our product roadmap
• Provide performance feedback to direct reports though Performance Reviews
• Mentor and Coach junior team members
• Advocate for prioritization of deliverables with the Data and Analytics Product and Project Managers
• Conduct review session with reports and the product owners
• Create incident reports in the rare event of a production outage

Operational support

• Administer and maintain data lake, data warehouse and data workflows for both the pre-production and production environments, ensuring that they are tuned for optimal performance and minimal downtime.
• Perform problem resolution and root cause analysis, along with recommending and implementing preventative techniques.
• Provide support for data platforms and environments to remote development teams.
• Provide time-critical support on production platforms via rotational on-call support.
• Support internal departments with ad-hoc client requests.

Security

• Protect Clickatell’s information, intellectual property, and corporate data systems in accordance with prescribed guidelines.
• Champion adherence to all relevant risk and compliance policies and procedures as communicated.
• Complete risk and compliance training as per internal requirements.
• Approve user access to data, which includes access grants based on the principle of least privilege.
• Audit user and system access to data lakes and data warehouses.
• Ensure adequate protection of the Clickatell brand and reputation.

Documentation

• Author and maintain documentation related to data engineering
• Actively participate in expanding existing documentation.

Review

• Peer reviews of code and queries created by team members.

Requirements of the Role

• A Diploma, B-Tech or bachelor's degree in engineering, computer science, information systems, informatics, data science or 10 years’ work experience in in relevant data management fields.

Work Experience

• 5+ years’ experience working with Big Data technologies.
• 7+ years’ experience coding with scripting languages, including Python and Unix/Linux Shell.
• Experience with cloud services and other technologies used at Clickatell
• AWS Databases: DynamoDB
• AWS Data Analytics: Kinesis, Athena (Presto DB), Elastic Map Reduce (EMR) or Hadoop/HDFS, Redshift, Glue
• AWS Compute: Lambda
• AWS Application Integration: Managed Apache Airflow
• AWS Storage: Simple Storage Service (S3)
• Other: Apache Hudi
• A background working in a high-volume payment transaction environment, or mobile technology platforms and systems integration would be advantageous – must be able to demonstrate a sustained track record of delivering high-quality outputs, on-time and to product or business specifications.
• Familiarity with Agile and Scrum methodologies and embracing of their values and principles.
• Familiarity with Version Control systems such as git (GitLab).

Advantageous – Knowledge and experience with the other technologies used at Clickatell:

• Databases: MySQL, PostgreSQL, Amazon RDS, Amazon Neptune
• Scripting Languages: Perl
• Productivity: JIRA, Confluence
• Monitoring: Datadog
• Familiarity with security and compliance standards:
• Exposure to data and security frameworks (ISO standards, NIST, etc)
• Exposure to compliance requirements around personal information (GDPR, POPIA, etc)

Proficient Knowledge and Ability

• Communicates effectively both verbally and in writing across multiple cultures and time zones.
• Assertiveness: ability to communicate effectively and explain a point of view convincingly.
• Teamwork: ability to work reliably and responsibly with internal and external colleagues to achieve departmental and organisational goals.
• Sound critical judgement to make good decisions, be someone the team can respect, trust, and depend on.
• Customer Focus: can identify and effectively respond to the diverse needs of development department’s stakeholders.
• Exceptional problem-solving skills, logical thinking, and analytical abilities, with a high level of attention to detail.
• Display foresight, with systematic and orderly planning.
• In-Depth Knowledge and Ability
• Advanced SQL development competency.
• Excellent technical documentation skills and can write clear and concise technical documents.