Senior Data Labeling Specialist (Nashville)

Senior Data Labeling Jobs in Nashville (Remote)

Rex.zone is hiring a Senior Data Labeling Specialist to own end-to-end data labeling and evaluation workflows for AI/ML training datasets across NLP, computer vision, and content safety. You will lead advanced annotation and reviewer-level QA, and run RLHF-style evaluations (preference ranking, rubric scoring, prompt evaluation) that connect label quality to model performance improvement.

What You Will Do

• Label and review complex text, image, and multimodal datasets while maintaining high inter-annotator agreement.
• Run QA evaluation programs: sampling plans, error taxonomy, root-cause analysis, and corrective actions tied to guideline compliance.
• Perform RLHF evaluation workflows such as pairwise preference ranking, rubric-based scoring, and prompt evaluation for instruction-following and safety behavior.
• Execute NLP labeling tasks including named entity recognition, classification, summarization evaluation, and groundedness/faithfulness checks.
• Execute computer vision annotation such as bounding boxes, polygons, keypoints, segmentation, and attribute tagging; validate edge cases.
• Support content safety labeling (e.g., harassment, self-harm, hate) with consistent escalation paths and audit trails.
• Partner with engineering/ML teams to translate model failure modes into updated guidelines, new label schemas, and targeted re-annotation.
• Document decisions, dataset specs, and QA findings to support reproducibility and dataset lineage.
  
  

Required Qualifications

• 3 years in data labeling/annotation or AI/ML evaluation, including QA evaluation and reviewer workflows.
• Hands-on experience with LLM evaluation, RLHF evaluation, or structured prompt evaluation using rubrics and preference judgments.
• Strong guideline compliance, error analysis, and understanding of how training data quality impacts model performance.
• Experience with NLP tasks (e.g., named entity recognition, text classification) and common dataset formats/tooling.
• Familiarity with computer vision annotation concepts and ambiguity resolution patterns.
• Ability to write clear guidelines, make consistent decisions, and communicate edge cases cross-functionally.
  
  

Remote & Location Notes

• Workplace: Remote, open to US-based candidates including Nashville, TN.
• May align to team collaboration hours for calibration and QA sign-off meetings.
  
  

Compensation

Competitive hourly base pay: $30–$50 per hour (USD).

Apply via Rex.zone to enter the matching workflow.