evalstone/evalscope/docs/en/advanced_guides/custom_dataset/embedding.md

Custom Text Retrieval Evaluation Dataset

Overview

When you need to evaluate the retrieval capabilities of Embedding models on domain-specific data (e.g., legal, medical, financial) or private datasets, you can build a custom dataset for evaluation. EvalScope, built on the MTEB framework, supports defining corpus, queries, and relevance judgments through simple JSONL files for end-to-end retrieval evaluation.

Step 1: Prepare Data

Directory Structure

Create a directory containing the following three JSONL files:

my_retrieval_data/
├── corpus.jsonl
├── queries.jsonl
└── qrels.jsonl

corpus.jsonl

The corpus file, with one JSON object per line representing a document to be retrieved.

Field	Type	Required	Description
_id	str	Yes	Unique document identifier
text	str	Yes	Document body content
title	str	No	Document title, used to assist retrieval

{"_id": "doc1", "text": "Climate change is leading to more extreme weather patterns.", "title": "Climate Change"}
{"_id": "doc2", "text": "The stock market surged today, led by tech stocks."}
{"_id": "doc3", "text": "Artificial intelligence is transforming industries by automating tasks and providing insights."}

queries.jsonl

The queries file, with one JSON object per line representing a retrieval query.

Field	Type	Required	Description
_id	str	Yes	Unique query identifier
text	str	Yes	Query text

{"_id": "query1", "text": "What are the impacts of climate change?"}
{"_id": "query2", "text": "What caused the stock market to rise today?"}
{"_id": "query3", "text": "How is artificial intelligence changing industries?"}

qrels.jsonl

The relevance judgments file, with one JSON object per line defining the relevance between a query and a document.

Field	Type	Required	Description
query-id	str	Yes	Corresponds to _id in queries.jsonl
corpus-id	str	Yes	Corresponds to _id in corpus.jsonl
score	int/float	Yes	Relevance score

{"query-id": "query1", "corpus-id": "doc1", "score": 1}
{"query-id": "query2", "corpus-id": "doc2", "score": 1}
{"query-id": "query3", "corpus-id": "doc3", "score": 1}

Score semantics:

• Binary (0/1): 1 means relevant, 0 means not relevant. Suitable for most retrieval scenarios.
• Graded relevance: Use multi-level scores (e.g., 0, 1, 2, 3), where higher values indicate greater relevance. Suitable for fine-grained evaluation that distinguishes "partially relevant" from "highly relevant".

Data Quality Recommendations

- The corpus should contain at least **100** documents and at least **50** queries for statistically meaningful evaluation results.
- Ensure each query has at least **1** relevant document annotation.
- When using binary labels, consider also annotating negative samples (score=0) to more accurately assess the model's discrimination ability.
- Include enough irrelevant "distractor" documents in the corpus to simulate real retrieval scenarios.

Step 2: Configure and Run Evaluation

Complete Configuration Example

from evalscope.run import run_task

task_cfg = {
    "work_dir": "outputs",
    "eval_backend": "RAGEval",
    "eval_config": {
        "tool": "MTEB",
        "models": [
            {
                "model_name_or_path": "AI-ModelScope/m3e-base",
                "pooling_mode": None,
                "max_seq_length": 512,
                "prompt": "",
                "model_kwargs": {"torch_dtype": "auto"},
                "encode_kwargs": {
                    "batch_size": 128,
                },
            }
        ],
        "eval": {
            "custom_tasks": [
                {
                    "name": "CustomRetrieval",
                    "data_path": "my_retrieval_data",
                }
            ],
            "overwrite_results": True,
            "limits": 500,
        },
    },
}

run_task(task_cfg=task_cfg)

Run Command

Save the above code as run_eval.py, then execute:

python run_eval.py

Interpreting Results

After evaluation completes, results are saved in the outputs/ directory. Key metrics include:

• NDCG@k: Normalized Discounted Cumulative Gain, measuring ranking quality (k=1,3,5,10)
• MAP@k: Mean Average Precision
• Recall@k: Recall rate
• Precision@k: Precision rate

Scores range from 0 to 1, with higher values indicating better retrieval performance.

Parameter Reference

Each task in the custom_tasks list corresponds to a CustomTaskConfig with the following fields:

Parameter	Type	Default	Description
name	str	"CustomRetrieval"	Task name, used to identify evaluation results
data_path	str	(required)	Dataset directory path, must contain corpus.jsonl, queries.jsonl, qrels.jsonl
eval_splits	List[str]	["test"]	Evaluation splits list

Other evaluation parameters (such as `models`, `limits`, `overwrite_results`, etc.) are consistent with the default configuration. See [MTEB Evaluation Parameter Reference](../../user_guides/backend/rageval_backend/mteb.md#parameter-explanation) for details.

FAQ

Field Format Errors

• Data files must be in JSONL format (one independent JSON object per line), not standard JSON array format.
• The _id field value must be a string type. Even numeric IDs should be written as "123" rather than 123.
• Field names in qrels.jsonl use hyphens: query-id and corpus-id, not underscores.

Impact of Insufficient Data

• When the corpus has too few documents (e.g., fewer than 10), metrics like Recall@10 may always be 1.0, making it impossible to effectively differentiate between models.
• Too few queries lead to high variance in evaluation results, lacking statistical significance.
• It is recommended to prepare at least hundreds of samples based on your actual business scenario.

Converting from Other Formats

If you already have data in TSV or CSV format, you can convert it to JSONL as follows:

import csv
import json

# Example: TSV/CSV to corpus.jsonl
with open("corpus.tsv", "r") as fin, open("corpus.jsonl", "w") as fout:
    reader = csv.DictReader(fin, delimiter="\t")  # For CSV, use delimiter=","
    for row in reader:
        obj = {"_id": row["id"], "text": row["text"]}
        fout.write(json.dumps(obj, ensure_ascii=False) + "\n")