What are the responsibilities and job description for the AI Evaluation Specialist - Remote position at YO AI Labs?
AI Evaluation Specialist
Role Type: Contractor
Location: Remote — US, Canada, UK, Ireland, Australia, New Zealand
We are seeking experienced AI Evaluation Specialists to assess and improve the quality of AI assistant outputs for an enterprise AI training project.
You'll evaluate AI-generated responses using defined quality standards, identify issues, and provide clear feedback to help improve AI systems. No prior formal AI experience is required.
Scope of Work- Evaluate AI-generated outputs against detailed rubrics for accuracy, relevance, quality, and adherence to guidelines.
- Apply consistent and objective judgment across large volumes of examples.
- Identify reasoning gaps, tool-use issues, logical errors, and inconsistencies.
- Provide concise, actionable feedback highlighting strengths and areas for improvement.
- Participate in discussions around rubric interpretation and evolving quality standards.
- Contribute to process improvements and evaluation best practices.
- Maintain accurate documentation of evaluations and recommendations.
- Experience in grading, quality assurance, editorial review, assessment, annotation, or similar analytical work.
- Advanced, daily use of AI assistants such as ChatGPT, Claude, or similar tools.
- Strong ability to analyze complex information and communicate findings clearly in writing.
- Experience with process improvement, rubric development, or quality assessment.
- Strong critical-thinking skills with a focus on consistency, accuracy, and fairness.
- Comfortable working independently on high-volume evaluation tasks.
- Strong attention to detail and ability to identify subtle quality issues.
- Collaborative approach to resolving ambiguous cases and improving evaluation criteria.
Salary : $30 - $90