Data Engineer at Cititec

About the Role:

• We are seeking an experienced Data Engineer to join our client's Data Engineering team. The ideal candidate will have a strong background in big data technologies, specifically Spark, Databricks, and Azure. You will be responsible for designing, building, and maintaining scalable data pipelines and architectures to support our data-driven initiatives.

Responsibilities:

• Design and implement data pipelines using Spark and Databricks on the Azure cloud platform to collect, store, process, and analyze large and complex data sets.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Develop and maintain data quality checks to ensure the accuracy and completeness of data.
• Optimize data pipelines to improve performance and efficiency in the Azure environment.
• Develop and implement data security and privacy policies and procedures in the context of Azure.
• Collaborate with data scientists and analysts to support their data needs.
• Continuously monitor and improve data pipeline processes to ensure they are functioning as expected in the Azure ecosystem.

Requirements:

• Bachelor's degree in Computer Science, Information Systems, or a related field.
• 3-4 years of experience in data engineering, specifically with Spark, Databricks, and Azure.
• Strong understanding of big data technologies and architecture patterns.
• Experience with data storage technologies such as Apache Cassandra, Amazon S3, and Azure Blob Storage.
• Strong programming skills in Scala or Python.
• Experience with data streaming technologies such as Apache Kafka and Azure Event Hubs.
• Strong understanding of SQL and database concepts.
• Excellent understanding of Azure services such as Azure Databricks, Azure Data Factory, and Azure Stream Analytics.
• Excellent problem-solving, critical thinking, and communication skills.