Prompt Engineer-Data Science & Quality Analysis

Checkmate is building advanced Voice AI systems for some of the largest restaurant and retail

brands in the US, including several in the top 10. Unlike many companies still in the prototype

phase, our AI solutions are live in production with real customers, achieving over 80%

accuracy. This is a $1 billion market opportunity, and we’re on track to scale to 3,000 stores by

the end of this year.

Join us at this pivotal moment of huge growth to shape AI products used daily by thousands of

staff and customers.

Essential Job Functions

• Prompt Design & Evaluation: Develop, test, and refine prompts for tasks such as text generation, question answering, data classification, and structured data extraction to optimize Voice AI performance.
• Data-Driven Analysis & Quality Measurement: Design evaluation frameworks and analyze prompt outputs using quantitative metrics, human-in-the-loop evaluation, and user feedback to identify improvement opportunities.
• Experimentation & Iteration: Conduct experiments to test prompt variations, measure their business and operational impact, and iterate to enhance accuracy, consistency, and safety.
• Regression Testing & Compliance: Build principled regression test suites using tools like LangFuse and Galileo to ensure prompts remain compliant and high performing as models and use cases evolve.
• Collaboration Across Teams: Work closely with data science, product, legal, engineering, and operations teams to align prompt designs with business goals, operational workflows, and compliance requirements.
• Model Adaptation & Strategy: Develop prompts across multiple LLMs (GPT, LLaMA, Gemini, and Checkmate’s fine-tuned models), understanding model differences to optimize outputs effectively.
• Team Leadership & Mentorship: To lead a team of analysts focused on prompt evaluation and data quality analysis, guiding prioritization, experimentation, and reporting. I collaborate with ops teams for seamless deployment and feedback loops.
• Research & Continuous Learning: Stay current on emerging prompting techniques, LLM behaviors, evaluation frameworks, and AI safety practices to keep Checkmate’s AI solutions best-in-class.
  
  

100% Remote

Salary $125,000 to $160,000

Requirements

• Bachelor’s degree in data science, Computer Science, Statistics, Engineering, or a related field with 3-6 years of experience in the relevant field.
• Strong analytical and data science skills, with hands-on experience in Python (pandas, NumPy, scikit-learn).
• Experience designing and conducting experiments and evaluations in applied AI or NLP contexts.
• Proficiency in SQL and working with relational databases (e.g., MySQL, PostgreSQL, Oracle, MS SQL).
• Good understanding of data processing, quality measurement, and testing fundamentals. Experience leading analyst or operations teams, with strong prioritization, mentorship, and collaboration skills.
• Strong problem-solving mindset with a drive to explore, optimize, and automate workflows. Excellent communication skills for presenting insights to technical and non-technical stakeholders.
• Candidates must be flexible and work during US hours at least until 6 pm ET in the USA, which is essential for this role & must also have their own system/work setup for remote work.
  
  

Preferred Qualifications

• Experience with LLM evaluation and prompt engineering workflows
• Familiarity with tools like LangFuse and Galileo for prompt evaluation and analysis.
• Knowledge of cloud platforms (AWS, GCP, Azure) and data pipeline tools
• Familiarity with machine learning concepts and NLP workflows
• Master’s or PhD in Data Science, Statistics, Computer Science, or a related field.
  
  

Benefits

• Health Care Plan (Medical, Dental & Vision)
• Retirement Plan (401k)
• Life Insurance (Basic, Voluntary & AD&D)
• Flexible Paid Time Off
• Family Leave (Maternity, Paternity)
• Short Term & Long Term Disability
• Training & Development
• Work From Home
• Stock Option Plan