Senior Java Engineer (Data Engineering) (CPT Hybrid) at Datafin Recruitment

ENVIRONMENT:

• WORK on cutting-edge Data Engineering projects of a US-based growing premium Software Product and Consulting Services company seeking a Senior Java Engineer with strong Data Engineering.
• You will be focused on executing critical enterprise data retention, migration, and privacy enhancement projects. This role combines robust Software Engineering capabilities with technical leadership skills to deliver production-quality solutions at petabyte scale.
• The successful candidate must have a Masters/Bachelors Degree in Computer Science/Engineering or related field with 10 years Data Engineering/Software Engineering experience with 4-7 years technical leadership.
• You will also need advanced proficiency in Scala or Java programming language with modern build systems (Gradle or Maven) & proven expertise in large-scale data migration and ETL pipeline development.

DUTIES:

Data Engineering & Migration Leadership –

• Design and implement large-scale data migration strategies for identity transformation across multi-petabyte datasets.
• Build comprehensive data lineage mapping and validation systems to ensure migration accuracy and data integrity.
• Develop scalable data transformation solutions supporting both batch and streaming processing with parallel pipeline orchestration.
• Implement advanced monitoring and observability solutions for data pipeline performance and system health tracking.
• Create testing and validation frameworks to ensure data accuracy and consistency during complex migrations.

Technical Leadership & Engineering Excellence –

• Provide technical leadership for data migration initiatives, taking responsibility for critical system components and project phases.
• Mentor Engineers through pair programming, code reviews, and collaborative data pipeline development.
• Lead technical design discussions for complex data migrations and create solutions meeting enterprise requirements.
• Demonstrate deep expertise in Data Engineering lifecycle management, from migration planning through deployment and operational support.
• Drive technical work forward by identifying and removing blockers, enabling continuous delivery of value.
• Collaborate effectively with stakeholders across disciplines, ensuring alignment and successful project outcomes.

REQUIREMENTS:

Qualifications –

• Masters or Bachelors Degree or equivalent experience in Computer Science, Engineering, or related field.

Experience/Skills –

• 10 Years Data Engineering/Software Engineering experience with 4-7 years technical leadership.
• Proven expertise in large-scale data migration and ETL pipeline development.
• Advanced proficiency in Scala or Java programming language with modern build systems (Gradle or Maven).
• Expertise in distributed file systems (HDFS) or cloud object storage (S3, GCS).
• Experience with cloud data platforms (AWS, GCP, Azure, or Databricks).
• Hands-on experience with AWS EKS (Kubernetes) for containerized data processing environments.
• Understanding of data privacy, security, and compliance requirements (GDPR experience preferred).
• Strong knowledge of data structures, algorithms, and distributed systems design principles.

Preferred to haves –

• Experience with Scala or Python for data processing applications.
• Experience with Apache Spark or PySpark for large-scale data processing and analytics.
• Experience with Hadoop ecosystem including HDFS, MapReduce, and related tools.
• Experience working in Enterprise Engineering environments with large-scale tools and development processes.
• Experience with data lineage analysis, dependency mapping, and data validation frameworks.
• Identity Management experience in large-scale distributed systems.
• Experience with modern data formats like Apache Iceberg, Parquet, ORC, or Avro.
• Proficiency in streaming platforms like Apache Kafka or AWS Kinesis.
• Hands-on experience with observability tools (Prometheus, Grafana, Spark UI).