Intermediate Data Engineer at Ignition Group

Responsibilities :

Data Ingestion & Processing

• Assist in building ETL/ELT pipelines using AWS services like AWS Glue, Lambda, and Step Functions.
• Work with Amazon S3 for data storage and ingestion.
• Support real-time and batch data processing jobs.

Data Storage & Management

• Load and manage data in AWS
• Work with Parquet, Avro, JSON, and CSV file formats in data lakes.

Data Transformation & Integration

• Support data transformations using PySpark, SQL, or AWS Glue ETL.
• Assist in writing SQL queries for data aggregation, cleansing, and structuring.

Automation & Orchestration

• Monitor scheduled workflows
• Write basic Python or Shell scripts for automating tasks.

Monitoring & Debugging

• Assist in monitoring and troubleshooting failed data pipelines.
• Perform unit testing and basic debugging of data workflows.

Education and training:

• Diploma in Computer Science, Information Technology, Engineering, or equivalent.
• Bachelor’s degree in computer science, Information Technology, Engineering, or equivalent advantageous.
• Relevant certification (Certified Data Engineer, Certificate Professional, etc).

Experience:

• 2 years in a similar role.
• Experience with architecture and design of secure, highly available and scalable systems.
• Experience with writing scripts using scripting language.
• Experience working with data sets and ordering data through various functions.
• Experience working with internet technologies.
• Experience dealing with internal and external stakeholders.
• Experience working with and/or on Power BI, Snowflake and AWS system advantageous.