ML Inference Engineer - Terminal-Bench Task Designer (vLLM / SGLang / TensorRT-LLM)

About the role

Toptal is looking for highly skilled ML inference and serving engineers to contribute to a specialized project called Terminal-Bench, which creates rigorous, real-world benchmarks for evaluating how well AI agents handle complex, expert-level tasks in a terminal environment. Compensation is tied to the challenges you design and successfully implement, so it is a task-based opportunity suited to engineers who enjoy architecting complex systems.

Key responsibilities

• Design complex challenges that turn real-world ML inference tasks into reproducible, command-line-solvable scenarios
• Package tasks using Docker and Git, documenting every dependency, data point and system requirement
• Write reference solutions and build automated testing suites that verify an agent reaches the correct result
• Create sophisticated validation scripts that detect subtle failures such as numerical drift or metric inaccuracies
• Iterate on designs based on feedback from the Toptal review team and performance data from AI agents

Required profile

• Significant hands-on experience in the ML space, beyond API calls or educational notebooks
• At least three years of active, hands-on experience in ML inference or serving, or a PhD paired with active, relevant research
• Ability to explain complex failure modes in ML pipelines, such as numerical drift or non-obvious metric bugs

Required skills

• Deep familiarity with PyTorch or JAX internals
• Direct experience with production-grade serving stacks such as vLLM, SGLang or TensorRT-LLM
• Strong command-line fluency and ability to manage complex file systems, databases and APIs in a production-like setting