Senior Data Annotation Specialist (Los Angeles) - Remote

Senior Data Annotation Jobs in Los Angeles (Remote, Full Time)

Rex.zone is hiring a Senior Data Annotation Specialist to lead high-quality training data workflows for AI/ML systems. You will support LLM training pipelines across NLP, RLHF preference data, prompt evaluation, content safety labeling, and computer vision annotation while partnering with engineering to improve model outcomes.

Key Responsibilities

• Lead senior annotation queues for LLM training pipelines and multimodal datasets
• Create and refine annotation guidelines, edge-case policies, and labeling taxonomies
• Run QA evaluation sampling, inter-annotator agreement checks, and error analysis
• Execute RLHF workflows (preference ranking, pairwise comparisons, rubric scoring)
• Perform prompt evaluation for helpfulness, harmlessness, and instruction following
• Deliver consistent NLP labels including NER and text classification
• Support computer vision annotation (bounding boxes, polygons, segmentation masks) as needed
• Coordinate with engineering/ops to resolve data issues, tooling gaps, and throughput bottlenecks
• Document decisions to ensure auditability for training data quality reviews
  
  

Required Qualifications

• Mid-senior experience in data labeling, data annotation, or AI data operations
• Expertise with annotation guidelines compliance and QA evaluation processes
• Familiarity with RLHF, LLM evaluation, and/or prompt evaluation methodologies
• Strong written communication for rubric design and reviewer feedback
• Comfort working with structured schemas for NLP tasks (NER, sentiment, intent, toxicity)
• Ability to work independently in a remote full-time environment
  
  

How To Apply

Apply through Rex.zone with a concise summary of your senior data annotation experience, task types shipped (RLHF, LLM evaluation, prompt evaluation, NER, computer vision), and examples of improving training data quality through QA evaluation and guideline refinement.