Data Engineer at Collinson

Purpose of the job

The Data Engineering Team is responsible for maintaining and enhancing the data platform, to deliver business value to both internal and external stakeholders.

As a Data Engineer, you’ll be a key member of this team which delivers functionality using the Agile delivery methodology. You will be responsible for designing, building, and maintaining data pipelines under the guidance of a Principal Software Engineer to the agreed development plan and to approved specifications.

You will have the opportunity to bring innovation to the team, using cutting-edge data engineering frameworks and techniques.

Key Responsibilities

• Design, develop, and maintain data pipelines for collecting, transforming, and loading data into various data stores
• Write efficient and scalable code in languages such as Python
• Develop collaborative relationships with internal stakeholders to understand their data requirements and provide solutions
• Resolve day-to-day challenges or issues without the need for technical guidance
• Provide competent, proactive, and professional technical support when required
• Deliver supporting documentation of a high quality
• Actively engage in and contribute to agile planning and design sessions
• Support agile process improvements that will improve productivity, effectiveness, and professionalism of the team, without impacting budgets and delivery plans
• Contribute to the Code Peer Review community to improve the quality of code produced and ensure it adheres to our standards
• Participate in regular team meetings to ensure the correct tasks are being worked on and share key decisions required, risks identified and highlight any issues encountered
• Support, Develop and Improve our CI/CD pipelines
• Maintain a good technical awareness and familiarity with new and forthcoming relevant Technologies
• Knowledge, skills and experience required

Required:

• University Degree in computer science, engineering, or related experience required
• At least 5 years’ experience in a Data Engineering role or with similar technical systems and technologies
• Experience with Scrum, Kanban, or another Agile application development methodology
• Experience of working across multiple delivery organisations practising distributed development

Specific Technical Skills:

• In-depth understanding of modern data architectures and data modelling practices
• Strong programming skills in Python, PySpark
• Extensive experience with AWS Technologies (Aurora, Glue, Lambda, Step Functions, S3, API Gateway, SQS, SNS, Athena)
• Familiarity with CI/CD for data pipelines using Terraform, CodeBuild and Flyway
• Usage of Source Control tools (GitHub Enterprise)