Custom Text Retrieval Evaluation Dataset#

Overview#

When you need to evaluate the retrieval capabilities of Embedding models on domain-specific data (e.g., legal, medical, financial) or private datasets, you can build a custom dataset for evaluation. EvalScope, built on the MTEB framework, supports defining corpus, queries, and relevance judgments through simple JSONL files for end-to-end retrieval evaluation.

Step 1: Prepare Data#

Directory Structure#

Create a directory containing the following three JSONL files:

my_retrieval_data/
├── corpus.jsonl
├── queries.jsonl
└── qrels.jsonl

corpus.jsonl#

The corpus file, with one JSON object per line representing a document to be retrieved.

Field	Type	Required	Description
`_id`	str	Yes	Unique document identifier
`text`	str	Yes	Document body content
`title`	str	No	Document title, used to assist retrieval

{"_id": "doc1", "text": "Climate change is leading to more extreme weather patterns.", "title": "Climate Change"}
{"_id": "doc2", "text": "The stock market surged today, led by tech stocks."}
{"_id": "doc3", "text": "Artificial intelligence is transforming industries by automating tasks and providing insights."}

queries.jsonl#

The queries file, with one JSON object per line representing a retrieval query.

Field	Type	Required	Description
`_id`	str	Yes	Unique query identifier
`text`	str	Yes	Query text

{"_id": "query1", "text": "What are the impacts of climate change?"}
{"_id": "query2", "text": "What caused the stock market to rise today?"}
{"_id": "query3", "text": "How is artificial intelligence changing industries?"}

qrels.jsonl#

The relevance judgments file, with one JSON object per line defining the relevance between a query and a document.

Field	Type	Required	Description
`query-id`	str	Yes	Corresponds to `_id` in queries.jsonl
`corpus-id`	str	Yes	Corresponds to `_id` in corpus.jsonl
`score`	int/float	Yes	Relevance score

{"query-id": "query1", "corpus-id": "doc1", "score": 1}
{"query-id": "query2", "corpus-id": "doc2", "score": 1}
{"query-id": "query3", "corpus-id": "doc3", "score": 1}

Score semantics:

Binary (0/1): 1 means relevant, 0 means not relevant. Suitable for most retrieval scenarios.
Graded relevance: Use multi-level scores (e.g., 0, 1, 2, 3), where higher values indicate greater relevance. Suitable for fine-grained evaluation that distinguishes “partially relevant” from “highly relevant”.

Data Quality Recommendations#

Tip

The corpus should contain at least 100 documents and at least 50 queries for statistically meaningful evaluation results.
Ensure each query has at least 1 relevant document annotation.
When using binary labels, consider also annotating negative samples (score=0) to more accurately assess the model’s discrimination ability.
Include enough irrelevant “distractor” documents in the corpus to simulate real retrieval scenarios.

Step 2: Configure and Run Evaluation#

Complete Configuration Example#

from evalscope.run import run_task

task_cfg = {
    "work_dir": "outputs",
    "eval_backend": "RAGEval",
    "eval_config": {
        "tool": "MTEB",
        "models": [
            {
                "model_name_or_path": "AI-ModelScope/m3e-base",
                "pooling_mode": None,
                "max_seq_length": 512,
                "prompt": "",
                "model_kwargs": {"torch_dtype": "auto"},
                "encode_kwargs": {
                    "batch_size": 128,
                },
            }
        ],
        "eval": {
            "custom_tasks": [
                {
                    "name": "CustomRetrieval",
                    "data_path": "my_retrieval_data",
                }
            ],
            "overwrite_results": True,
            "limits": 500,
        },
    },
}

run_task(task_cfg=task_cfg)

Run Command#

Save the above code as run_eval.py, then execute:

python run_eval.py

Interpreting Results#

After evaluation completes, results are saved in the outputs/ directory. Key metrics include:

NDCG@k: Normalized Discounted Cumulative Gain, measuring ranking quality (k=1,3,5,10)
MAP@k: Mean Average Precision
Recall@k: Recall rate
Precision@k: Precision rate

Scores range from 0 to 1, with higher values indicating better retrieval performance.

Parameter Reference#

Each task in the custom_tasks list corresponds to a CustomTaskConfig with the following fields:

Parameter	Type	Default	Description
`name`	str	`"CustomRetrieval"`	Task name, used to identify evaluation results
`data_path`	str	(required)	Dataset directory path, must contain `corpus.jsonl`, `queries.jsonl`, `qrels.jsonl`
`eval_splits`	List[str]	`["test"]`	Evaluation splits list

Note

Other evaluation parameters (such as models, limits, overwrite_results, etc.) are consistent with the default configuration. See MTEB Evaluation Parameter Reference for details.

FAQ#

Field Format Errors#

Data files must be in JSONL format (one independent JSON object per line), not standard JSON array format.
The _id field value must be a string type. Even numeric IDs should be written as "123" rather than 123.
Field names in qrels.jsonl use hyphens: query-id and corpus-id, not underscores.

Impact of Insufficient Data#

When the corpus has too few documents (e.g., fewer than 10), metrics like Recall@10 may always be 1.0, making it impossible to effectively differentiate between models.
Too few queries lead to high variance in evaluation results, lacking statistical significance.
It is recommended to prepare at least hundreds of samples based on your actual business scenario.

Converting from Other Formats#

If you already have data in TSV or CSV format, you can convert it to JSONL as follows:

import csv
import json

# Example: TSV/CSV to corpus.jsonl
with open("corpus.tsv", "r") as fin, open("corpus.jsonl", "w") as fout:
    reader = csv.DictReader(fin, delimiter="\t")  # For CSV, use delimiter=","
    for row in reader:
        obj = {"_id": row["id"], "text": row["text"]}
        fout.write(json.dumps(obj, ensure_ascii=False) + "\n")