AI EVAL Engineer

Akkodis is seeking an AI EVAL Engineer for a Contract with a client in Bellevue, WA(Remote). Candidates must have strong Python programming skills and hands-on experience with AI evaluation frameworks and metrics in a Linux environment.

Rate Range: $50/hour to $53/hour; The rate may be negotiable based on experience, education, geographic location, and other factors.

AI EVAL Engineer Job Responsibilities Include

• Design, implement, and automate evaluation test suites to measure LLM accuracy, relevance, safety, latency, and cost across zero-shot, few-shot, and system-prompt scenarios.
• Define and apply robust evaluation metrics (e.g., precision/recall, BLEU/ROUGE, F1, hallucination rate, throughput, cost-per-output) and establish reproducible baselines for model comparison.
• Build datasets, ground-truth references, and benchmarks, and maintain versioned test cases for consistent, repeatable scoring.
• Develop batch evaluation pipelines in Python (and other languages as needed) with API integrations, integrating frameworks like OpenAI Evals, HuggingFace evals, Promptfoo, Ragas, DeepEval, or LM Eval Harness.
• Conduct performance benchmarking and analysis across Azure OpenAI (and other providers), reporting insights on speed, scalability, and resource efficiency.
• Assess and mitigate AI safety, bias, and hallucination risks, while collaborating with product, research, and platform teams to improve prompts, guardrails, and overall model quality.
  
  

Required Qualifications

• Bachelor’s or master’s in computer science, Data Science, AI/ML, or related field.
• 3–5 years in AI/ML evaluation, benchmarking, or applied ML (including LLMs and generative AI).
• Strong Python skills with hands-on experience in evaluation frameworks (e.g., OpenAI Evals, Hugging Face evals, Promptfoo, Ragas, DeepEval, LM Eval Harness) and defining/applying metrics (precision/recall, BLEU/ROUGE, F1, hallucination rate, latency, cost).
• Practical experience with Azure OpenAI (and/or OpenAI/Anthropic/Google AI), test automation pipelines, and benchmarking across zero-/few-shot prompts; familiarity with RAG evaluation and AI safety/bias testing is a plus.
  
  

If you are interested in this role, then please click APPLY NOW. For other opportunities available at Akkodis, or any questions, feel free to contact me at Shubham.Vasishtha@akkodisgroup.com.

Pay Details: $50.00 to $53.00 per hour

Benefit offerings available for our associates include medical, dental, vision, life insurance, short-term disability, additional voluntary benefits, EAP program, commuter benefits and a 401K plan. Our benefit offerings provide employees the flexibility to choose the type of coverage that meets their individual needs. In addition, our associates may be eligible for paid leave including Paid Sick Leave or any other paid leave required by Federal, State, or local law, as well as Holiday pay where applicable.

Equal Opportunity Employer/Veterans/Disabled

Military connected talent encouraged to apply

To read our Candidate Privacy Information Statement, which explains how we will use your information, please navigate to https://www.akkodis.com/en/us/candidate-privacy-policy

Requirements

The Company will consider qualified applicants with arrest and conviction records in accordance with federal, state, and local laws and/or security clearance requirements, including, as applicable:

• The California Fair Chance Act
• Los Angeles City Fair Chance Ordinance
• Los Angeles County Fair Chance Ordinance for Employers
• San Francisco Fair Chance Ordinance
  
  

Massachusetts Candidates Only: It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.