Job Opportunities at Black Swan Data

Job Summary:

•  As an Associate Data Engineer, you will assist in building and maintaining our data infrastructure, working closely with our data engineering team to optimize our data systems and ensure data quality and accessibility. This role provides an exciting opportunity to grow your skills and career in data engineering, with extensive training and mentorship provided.

Key Responsibilities

• Data Infrastructure Support
• Assist in building and maintaining data pipelines, integrating data from various sources.
• Support the maintenance and optimization of data storage solutions, ensuring data integrity and accessibility.
• Help in implementing data validation procedures to ensure data accuracy and consistency.
• Collaboration and Communication
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Participate in team meetings and contribute to planning and problem-solving discussions.
• Continuous Learning and Improvement
• Engage in training and educational opportunities to enhance your data engineering skills.
• Contribute to continuous improvement efforts by identifying opportunities to optimize data systems and processes.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent experience in a Data Analyst or similar role.
• Basic understanding of data engineering concepts and principles.
• Familiarity with a programming language (Python, Java, etc.) is beneficial.
• Good problem-solving skills and a willingness to learn.
• Excellent communication and collaboration skills.

Preferred Qualifications

• Experience with SQL or other database technologies.
• Knowledge of data visualization tools (e.g., Tableau, Power BI) is a plus.

go to method of application »

Job Summary:

•  The Principal Data Engineer will play a critical role in designing, developing, and optimizing our data infrastructure, pipelines, and processing systems. This role requires strong technical expertise in Scala, extensive experience building and maintaining data pipelines, and a deep understanding of NLP techniques. The successful candidate will work closely with data scientists, analysts, and other stakeholders to ensure the seamless integration and accessibility of data across the organization.

Key Responsibilities

• Data Infrastructure and Pipeline Development
• Design, build, and maintain scalable and reliable data pipelines to ingest, process, and store large volumes of structured and unstructured data.
• Develop and optimize data processing systems using Scala and other programming languages, as required.
• Implement and manage data storage solutions, ensuring data integrity, security, and accessibility.
• NLP and Advanced Analytics
• Collaborate with data scientists to develop and implement NLP models and algorithms for text analysis, sentiment analysis, and other advanced analytics use cases.
• Optimize NLP pipelines for performance, scalability, and maintainability.
• Stay current with the latest research and developments in NLP and incorporate new techniques as appropriate.
• Data Quality and Governance
• Establish and enforce data quality standards, data governance policies, and best practices.
• Implement data validation, monitoring, and anomaly detection mechanisms to ensure data accuracy and consistency.
• Work closely with cross-functional teams to address data quality issues and drive continuous improvement.
• Technical Leadership and Collaboration
• Serve as a subject matter expert on Scala, data pipelines, and NLP within the organization.
• Mentor and guide junior data engineers in best practices, technical skills, and career development.
• Collaborate with data scientists, analysts, and other stakeholders to define requirements, provide technical guidance, and support data-driven decision-making.

Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent work experience).
• Extensive experience in data engineering, with a strong focus on Scala, data pipelines, and NLP.
• Deep understanding of distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop) and database technologies (e.g., SQL, NoSQL, columnar storage).
• Experience with cloud platforms and services (e.g., AWS, GCP, Azure) and containerization (e.g., Docker, Kubernetes).
• Knowledge of NLP libraries and tools (e.g., spaCy, NLTK, Gensim) and machine learning frameworks (e.g., TensorFlow, PyTorch).
• Strong problem-solving, analytical, and communication skills.

Preferred Qualifications

• Master's or Ph.D. in a related field.
• Experience with other programming languages (e.g., Python, Java) and big data technologies (e.g., Kafka, HBase, Cassandra).