AI Platform Engineer - Technology R&D at Signant Health

About the Role

• We are seeking an AI Engineer to build and optimize Signant Health's AI Platform and agents that automate workflows across our organization. This role sits at the intersection of application development and model optimization, requiring both strong software engineering fundamentals and expertise in generative AI technologies. You'll be responsible for developing and optimizing our AI platform components, developing intelligent agents, implementing RAG systems, and ensuring high-quality, secure, and compliant AI responses through rigorous evaluation frameworks and robust guardrails. 
• This position offers the opportunity to shape how AI transforms our organization's operations, working on production systems that directly impact efficiency and innovation across multiple teams while maintaining the highest standards of data security, privacy, and regulatory compliance. 

What you will do:

Application Development 

• Design and implement AI-powered agents that automate complex, multi-step workflows across the organization.
• Build RAG (Retrieval-Augmented Generation) systems using vector stores and knowledge bases to ground AI responses in organizational knowledge. 
• Develop prompt engineering strategies and context optimization techniques to maximize accuracy and reliability of AI outputs.
• Create integrations between AI systems and existing tools, APIs, and data sources. 
• Implement memory systems and state management for conversational agents and long-running workflows.

Platform Optimization & Evaluation 

• Develop comprehensive evaluation frameworks to measure accuracy, relevance, and quality of AI system responses.
• Design and execute experiments to optimize agent performance, including A/B testing of different prompting strategies. 
• Monitor production AI systems and implement improvements based on performance metrics and user feedback. 
• Optimize inference costs and latency while maintaining quality standards. 
• Build tooling and dashboards for observability into AI system behavior. 

Model Development & Fine-Tuning 

• Engineer datasets for training and fine-tuning language models on organization-specific tasks.
• Implement and evaluate fine-tuning pipelines for LLMs and SLMs (Small Language Models). 
• Optimize model inference for production environments, balancing speed, cost, and quality. 
• Conduct experiments to determine when fine-tuning, prompt engineering, or RAG approaches are most appropriate.
• Stay current with advances in language models and evaluate new models for potential organizational use.

Security & Compliance 

• Design and implement guardrails and content filtering mechanisms to prevent harmful or inappropriate AI outputs. 
• Develop hallucination detection and mitigation strategies to ensure factual accuracy and reliability of agent responses. 
• Establish security policies for AI systems including data access controls, prompt injection prevention, and sensitive information handling.
• Create compliance frameworks that align AI systems with regulatory requirements (GDPR, HIPAA, SOC2, etc.) and industry standards. 
• Implement monitoring and alerting systems to detect anomalous AI behavior, security incidents, or compliance violations.
• Conduct AI safety assessments and red-teaming exercises to identify vulnerabilities and edge cases. 
• Document AI decision-making processes and maintain audit trails for regulatory review and internal governance. 
• Collaborate with security, legal, and compliance teams to ensure AI systems meet organizational risk management standards.

Collaboration & Technical Leadership 

• Partner with business stakeholders to identify high-impact automation opportunities. 
• Document AI system architectures, design decisions, and best practices. 
• Contribute to technical standards and guidelines for AI development across the organization.
• Mentor team members on AI technologies and share knowledge through internal presentations or documentation. 
• Participate in code reviews and provide feedback on AI system designs. 

Preferred Qualifications:

Education 

• AWS AI/ML certifications are preferred. 

Experience:

Required: 

• 5+ years of software engineering experience with strong proficiency in Python and production system development.
• 2+ years of hands-on experience building and deploying AI/ML applications in production environments.
• Demonstrated expertise in prompt engineering, context engineering, and optimizing LLM-based applications for accuracy and performance. 
• Experience implementing RAG (Retrieval-Augmented Generation) systems using vector databases (PostgresSQL (pgvector), Amazon Bedrock Knowledge Bases, MongoDB, Pinecone). 
• Proven track record of building evaluation frameworks and metrics to measure AI system quality, accuracy, and reliability.
• Experience with LLM fine-tuning, dataset engineering, and model evaluation methodologies.
• Strong understanding of AI security principles including guardrails, content filtering, hallucination detection, and prompt injection prevention. 
• Experience working with APIs, microservices architectures, and distributed systems. 
• Proficiency with version control (Git), CI/CD pipelines, and modern software development practices. 

Preferred: 

• Experience building autonomous agents or multi-step AI workflows with tool use and decision-making capabilities. 
• Knowledge of inference optimization techniques, model quantization, or deployment optimization for LLMs and SLMs. 
• Familiarity with regulatory compliance frameworks (HIPAA, GDPR, SOC2) in the context of AI systems. 
• Experience implementing monitoring, logging, and alerting systems for production AI applications. 
• Background in healthcare, life sciences, or clinical trial operations. 
• Contributions to open-source AI projects or published work in applied AI/ML. 
• Experience with AI safety, red-teaming, or adversarial testing methodologies. 
• Familiarity with modern AI frameworks (Strands, LangChain, CrewAI, LlamaIndex) and model providers (Amazon Bedrock, Anthropic, OpenAI, Gemini).