AI Evaluation Specialist – Remote Contract

About the role

We are seeking a detail‑oriented AI Evaluation Specialist to design, execute, and refine evaluation tasks that benchmark AI performance in real‑world computer‑based workflows. This contract position is fully remote and offers flexible hours ranging from 10 to 40 per week.

Key responsibilities

• Design self‑contained evaluation tasks, including prompts, supporting files, and grading rubrics.
• Define clear, unambiguous criteria for successful and unsuccessful task outcomes across diverse administrative scenarios.
• Observe AI agent behavior, document observations, and produce concise, high‑quality English reports.
• Iterate and improve evaluation tasks and rubrics based on feedback and team collaboration.
• Collaborate with the client’s team to share insights and drive continuous improvement in AI evaluation techniques.

Required profile

• Minimum experience in roles that demand precise written communication and structured thinking (e.g., paralegal, executive assistant, junior analyst, librarian, research assistant, technical writer, QA analyst).
• Native or fluent English writing ability with a track record of producing succinct, specific, and unambiguous observations.
• Proven experience designing or applying rubric‑based evaluation, grading against set criteria, or building structured scoring frameworks.
• High attention to detail and ability to spot subtle patterns or inconsistencies.
• Excellent written and verbal communication skills for documenting nuanced feedback.

Required skills

What we offer

• Competitive hourly rate ranging from $20 to $35.
• Remote work with flexible scheduling.
• Opportunity to influence AI performance benchmarking for a leading client.