Senior ETL Engineer II at Impact

Why this role is exciting: 

• The Senior ETL Engineer is able to manage, process and analyze large datasets using big data technologies such as Spark, SingleStore and BigQuery. The ideal candidate will be responsible for designing and implementing efficient ETL (Extract, Transform, Load) processes to move, transform, and aggregate data from various sources into a centralized data warehouse and purpose-built data marts. The Senior ETL engineer will maintain ETL code and the resulting data assets, ensuring internal data quality and integrity standards are adhered to. 
• The position requires a strong analytical mindset, attention to detail and experience with big data technologies. This is a highly collaborative role as the ETL engineer needs to engage with Subject Matter Experts to implement business logic, understand source data structures and ensure data outputs are accurate, fit-for-purpose, pass quality assurance and provide value to the business.

Responsibilities: 

• Design, develop and maintain ETL processes leveraging big data technologies such as Apache Spark, Google DataFlow, SingleStore, Impala, Kudu and BigQuery
• Write and maintain Pyspark ETL notebooks in Databricks
• Assist Business Intelligence engineers, Data Scientists and Data Analysts with extraction and visualization of produced data sets
• Work with data scientists, data engineers, subject matter experts (SMEs) and other stakeholders to gather requirements and understand the data needs of the organization
• Develop and maintain data models (ERDs), data dictionaries and data flow diagrams
• Document business logic used in ETL processes
• Maintain table and column metadata
• Build data pipelines and ETL workflows to extract data from various sources, transform it to meet business requirements and load it into a centralized data warehouse and/or purpose-build data marts
• Identify and contribute to common utility libraries used by data teams
• Ensure data quality and integrity of ETL processes and troubleshoot any issues that arise
• Monitor ETL performance and reliability and optimize processes for efficiency and scalability
• Optimize total system performance related to ETL workloads - ensure efficient use of compute resources and stability of data systems
• Optimize existing data pipelines for simplicity and efficiency.
• Conduct periodic integrity checks on productionalized data assets
• Safeguard sensitive company data
• Work with QA to extend and enhance programmatic validation of productionalized data assets
• Stay up-to-date with the latest big data technologies and best practices
• Assist in automating manual ETL and data management processes
• Approve & Merge Pull Requests
• Mentor and train junior colleagues
• Actively participate in the hiring process and performance management of team members

Does this sound like you?

• Bachelor's or Master's degree in Computer Science, Data Science or related field
• 6+ years of experience in ETL development and data warehousing using big data technologies such as Apache Spark, Google DataFlow, SingleStore, Impala, Kudu and/or BigQuery
• Proven track record in developing enterprise-level data marts
• Experience with Databricks advantageous
• Experience with Google Cloud Platform and BigQuery advantageous
• Strong knowledge of SQL and relational database management systems
• Python programming skills required
• Strong data modeling and schema design experience
• Experience with workflow management tools such as Airflow, Luigi or Oozie advantageous
• Knowledge of data integration patterns, data load patterns and best practices required
• Strong analytical and problem-solving skills
• Excellent written and verbal communication skills
• Good leadership and workload management skills advantageous
• Ability to work in a team environment and collaborate with other stakeholders