What are the responsibilities and job description for the Data Scientist – Conversational AI & GenAI Evaluation position at ClifyX?
Job Title: Data Scientist – Conversational AI & GenAI Evaluation
Work Location : Johnston, RI or Westwood, MA (Hybrid Onsite)
Contract duration: 12 months Contract
Detailed job description - Skill Set:
We are seeking an experienced Data Scientist to support the development and evaluation of AI-powered fraud self-service voice agents and conversational AI systems. The primary responsibility is not model deployment or engineering implementation, but designing evaluation frameworks, measuring system performance, identifying failure patterns, conducting root-cause analysis, and optimizing model behavior through data-driven experimentation.
Key Responsibilities
- Design and execute evaluation frameworks for LLM, RAG, and multi-turn conversational AI systems.
- Develop metrics to assess customer intent recognition, conversation quality, guardrail effectiveness, and business outcomes.
- Analyze voice-agent interactions and identify areas of failure, drift, and performance degradation.
- Perform prompt tuning and experimentation to improve model accuracy and reliability.
- Conduct root-cause analysis of conversational failures and recommend remediation strategies.
- Measure performance across different model configurations, prompts, and guardrail implementations.
- Partner with AI Engineering and Product teams to validate solutions before production deployment.
- Build dashboards and reports that communicate model effectiveness and operational impact.
- Support fraud-related customer service use cases, including intent detection and multi-turn conversation flows.
Success Criteria
- Develop reliable evaluation methodologies for conversational AI systems.
- Quantify the effectiveness of fraud self-service voice agents.
- Optimize prompts, retrieval strategies, and guardrails using empirical evidence.
- Deliver actionable insights that improve customer experience and model performance.
- Establish measurable KPIs for intent detection and multi-turn conversation success.
Must have:
- Strong background in Data Science, Machine Learning, Generative AI, or a related quantitative field.
- Hands-on experience evaluating LLM, RAG, Agentic AI, or Conversational AI solutions.
- Deep understanding of model evaluation techniques and metrics, including:
- Precision@K
- Recall@K
- Mean Reciprocal Rank (MRR)
- F1 Score
- Retrieval and generation quality assessment
- Experience performing experimentation, statistical analysis, and performance benchmarking.
- Strong Python programming skills.
- Experience with machine learning libraries and frameworks such as Scikit-learn, XGBoost, Pandas, NumPy, and related tools.
- Ability to communicate technical findings succinctly to highly technical stakeholders.
Desired Skills:
- Experience with:
- Generative AI and LLM ecosystems
- Multi-agent systems
- RAG/Agentic RAG architectures
- Amazon Bedrock
- AWS SageMaker
- Databricks
- MLflow
- LangSmith
- Weights & Biases
- Knowledge of conversational AI, IVR systems, digital assistants, and voice agents.
- Experience in financial services, fraud detection, or customer service automation.
Important Note
This role is primarily a Data Science and AI Evaluation position, not an AI Engineering or deployment-focused role. The emphasis is on measuring, analyzing, validating, and improving AI system performance rather than building production deployment pipelines.