evalstone/evalscope/docs/en/benchmarks/researchrubrics.md

ResearchRubrics

Overview

ResearchRubrics evaluates Deep Research agents on realistic, open-ended research tasks. Each task pairs a user prompt with expert-written, fine-grained rubrics covering explicit and implicit requirements, information synthesis, references, communication quality, and instruction following.

Task Description

• Task Type: Multi-turn research agent / long-form report generation
• Input: One open-ended research prompt
• Output: A Markdown research report produced after iterative tool use
• Dataset: 101 tasks and 2,593 weighted rubric criteria
• Metric: Binary rubric compliance score

Key Features

• 101 open-ended Deep Research tasks paired with 2,593 expert-written, weighted rubric criteria.
• Rubrics cover explicit and implicit requirements, information synthesis, references, communication quality, and instruction following, and each is graded independently.
• Negative-weight criteria capture undesirable behaviours and subtract from the score when present.
• Long reports are graded with the official chunk-evidence-synthesis procedure once they exceed the configured judge context threshold.

Agent Environment

• Uses EvalScope's built-in agent environment by default and does not require agent_config. The agent can use bash to access the network, gather information, and produce a final report.
• The default environment uses the host network and a temporary working directory, but does not provide complete filesystem isolation. Do not run untrusted models on shared or sensitive machines.
• The default strategy is function_calling with a 50-step limit. Use NativeAgentConfig to override the strategy or step limit; react is also available. Both strategies require native function calling support.
• Add dedicated search or web-fetching tools through NativeAgentConfig, or use ExternalAgentConfig to run the task with another agent framework.
• When the step limit is reached, the model is asked to produce a final report from the information already collected so the result can still be reviewed and scored.

Evaluation Notes

• ResearchRubrics requires judge.models and judge.strategy='auto' or 'llm'. Gemini 2.5 Pro is the recommended judge for comparison with the paper, but no provider or model is hard-coded.
• Every rubric is graded independently as Satisfied (1) or Not Satisfied (0), matching the public binary grader. The paper's ternary scores are not directly comparable.
• Negative-weight criteria subtract from the numerator when the undesirable behavior is present. Scores are not clipped.
• Long reports are evaluated with the official chunk-evidence-synthesis approach when they exceed the configured judge context threshold.
• A full run performs 2,593 rubric evaluations and can be expensive. Current-events tasks are also sensitive to the date and web sources available at evaluation time.

Configuration

• judge_context_limit: 150,000 estimated tokens
• judge_chunk_size: 100,000 estimated tokens

The judge must be configured explicitly. For example:

from evalscope import TaskConfig, run_task

run_task(TaskConfig(
    model='YOUR_AGENT_MODEL',
    datasets=['researchrubrics'],
    judge={
        'strategy': 'llm',
        'models': {
            'model_id': 'YOUR_JUDGE_MODEL',
            'api_url': 'OPENAI_COMPATIBLE_JUDGE_URL',
            'api_key': 'YOUR_JUDGE_API_KEY',
            'generation_config': {'temperature': 0.0},
        },
    },
    limit=1,
))

Resources: Paper | GitHub | Dataset

Properties

Property	Value
Benchmark Name	researchrubrics
Dataset ID	evalscope/researchrubrics
Paper	Paper
Tags	Agent, MultiTurn, Reasoning, Retrieval
Metrics	compliance_score
Default Shots	0-shot
Evaluation Split	train

Data Statistics

Metric	Value
Total Samples	101
Prompt Length (Mean)	555.35 chars
Prompt Length (Min/Max)	102 / 1747 chars

Sample Example

Subset: default

{
  "input": [
    {
      "id": "f1132c85",
      "content": "I want to create a plan for July 4, 2025, i.e., Independence Day in Washington DC. I would like an itinerary of all the things to do and all the activities that are planned for Independence Day. Create a plan for the whole day and also extend it to the weekend, if required. Provide some reviews or explain why one should visit the place or engage in the activity. Add any additional information that is required."
    }
  ],
  "target": "[{\"criterion\": \"The response covers the period from 9:00 AM or earlier through at least 10:00 PM on 4 July 2025\", \"weight\": 5.0, \"axis\": \"Explicit Criteria\"}, {\"criterion\": \"The response contains clear section headers for parts of the schedul ... [TRUNCATED 4470 chars] ...  events from years other than 2025 (e.g., seeing a miltiary parade, information about \\\"A Capital Fourth\\\" for 2024, information the parade route for 2023, referencing a cancellation from 2020).\", \"weight\": -4.0, \"axis\": \"Explicit Criteria\"}]",
  "id": 0,
  "group_id": 0,
  "tools": [
    {
      "name": "bash",
      "description": "Execute a bash command inside the sandbox environment. Returns the combined stdout / stderr output of the command.",
      "parameters": {
        "properties": {
          "command": {
            "type": "string",
            "description": "The bash command to execute."
          },
          "timeout": {
            "type": "number",
            "description": "Maximum execution time in seconds (default: 60).",
            "default": 60
          }
        },
        "required": [
          "command"
        ]
      }
    }
  ],
  "metadata": {
    "sample_id": "6847465956a0f6376a605427",
    "domain": "Current Events",
    "conceptual_breadth": "Simple",
    "logical_nesting": "Intermediate",
    "exploration": "Medium"
  }
}

Note: Some content was truncated for display.

Prompt Template

Prompt Template:

{question}

Extra Parameters

Parameter	Type	Default	Description
judge_context_limit	int	150000	Estimated token limit before rubric judging switches to chunking.
judge_chunk_size	int	100000	Maximum estimated tokens in each document chunk sent to the judge.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets researchrubrics \
    --agent-config '{"mode":"native","strategy":"function_calling","max_steps":50}' \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import TaskConfig, run_task
from evalscope.api.agent import NativeAgentConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['researchrubrics'],
    agent_config=NativeAgentConfig(
        strategy='function_calling',
        max_steps=50,
    ),
    dataset_args={
        'researchrubrics': {
            # extra_params: {}  # uses default extra parameters
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)