{
  "meta": {
    "pretty_name": "WMT2024++",
    "dataset_id": "extraordinarylab/wmt24pp",
    "paper_url": null,
    "tags": [
      "MultiLingual",
      "MachineTranslation"
    ],
    "metrics": [
      {
        "bleu": {}
      },
      {
        "bert_score": {
          "model_id_or_path": "AI-ModelScope/xlm-roberta-large",
          "model_type": "xlm-roberta-large"
        }
      },
      {
        "comet": {
          "model_id_or_path": "evalscope/wmt22-comet-da"
        }
      }
    ],
    "few_shot_num": 0,
    "eval_split": "test",
    "train_split": "",
    "subset_list": [
      "en-ar_eg",
      "en-ar_sa",
      "en-bg_bg",
      "en-bn_in",
      "en-ca_es",
      "en-cs_cz",
      "en-da_dk",
      "en-de_de",
      "en-el_gr",
      "en-es_mx",
      "en-et_ee",
      "en-fa_ir",
      "en-fi_fi",
      "en-fil_ph",
      "en-fr_ca",
      "en-fr_fr",
      "en-gu_in",
      "en-he_il",
      "en-hi_in",
      "en-hr_hr",
      "en-hu_hu",
      "en-id_id",
      "en-is_is",
      "en-it_it",
      "en-ja_jp",
      "en-kn_in",
      "en-ko_kr",
      "en-lt_lt",
      "en-lv_lv",
      "en-ml_in",
      "en-mr_in",
      "en-nl_nl",
      "en-no_no",
      "en-pa_in",
      "en-pl_pl",
      "en-pt_br",
      "en-pt_pt",
      "en-ro_ro",
      "en-ru_ru",
      "en-sk_sk",
      "en-sl_si",
      "en-sr_rs",
      "en-sv_se",
      "en-sw_ke",
      "en-sw_tz",
      "en-ta_in",
      "en-te_in",
      "en-th_th",
      "en-tr_tr",
      "en-uk_ua",
      "en-ur_pk",
      "en-vi_vn",
      "en-zh_cn",
      "en-zh_tw",
      "en-zu_za"
    ],
    "description": "\n## Overview\n\nWMT2024++ is a comprehensive machine translation benchmark based on the WMT 2024 news translation task. It supports 54 language pairs with English as the source language, enabling evaluation of translation quality across diverse target languages.\n\n## Task Description\n\n- **Task Type**: Machine Translation\n- **Input**: Source text in English with translation prompt\n- **Output**: Translated text in the target language\n- **Language Pairs**: 54 pairs (English to 54 target languages)\n\n## Key Features\n\n- Extensive multilingual coverage (54 target languages)\n- News domain text for real-world applicability\n- Multiple evaluation metrics (BLEU, BERTScore, COMET)\n- Standardized prompt template for consistent evaluation\n- Supports batch scoring for efficiency\n\n## Evaluation Notes\n\n- Default configuration uses **0-shot** evaluation\n- Metrics: **BLEU**, **BERTScore** (XLM-RoBERTa), **COMET** (wmt22-comet-da)\n- Evaluates on **test** split\n- Language-specific normalization applied\n- COMET metric requires `unbabel-comet` package\n- Subsets represent individual language pairs (e.g., `en-zh_cn`, `en-de_de`)\n",
    "prompt_template": "Translate the following {source_language} sentence into {target_language}:\n\n{source_language}: {source_text}\n{target_language}:",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {},
    "sandbox_config": {},
    "category": "llm"
  },
  "statistics": {
    "total_samples": 52800,
    "subset_stats": [
      {
        "name": "en-ar_eg",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 156.45
      },
      {
        "name": "en-ar_sa",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 156.26
      },
      {
        "name": "en-bg_bg",
        "sample_count": 960,
        "prompt_length_mean": 269.26,
        "prompt_length_min": 81,
        "prompt_length_max": 1045,
        "prompt_length_std": 174.62,
        "target_length_mean": 192.87
      },
      {
        "name": "en-bn_in",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 186.85
      },
      {
        "name": "en-ca_es",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 206.18
      },
      {
        "name": "en-cs_cz",
        "sample_count": 960,
        "prompt_length_mean": 261.26,
        "prompt_length_min": 73,
        "prompt_length_max": 1037,
        "prompt_length_std": 174.62,
        "target_length_mean": 187.1
      },
      {
        "name": "en-da_dk",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 202.08
      },
      {
        "name": "en-de_de",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 223.22
      },
      {
        "name": "en-el_gr",
        "sample_count": 960,
        "prompt_length_mean": 261.26,
        "prompt_length_min": 73,
        "prompt_length_max": 1037,
        "prompt_length_std": 174.62,
        "target_length_mean": 219.29
      },
      {
        "name": "en-es_mx",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 213.41
      },
      {
        "name": "en-et_ee",
        "sample_count": 960,
        "prompt_length_mean": 267.26,
        "prompt_length_min": 79,
        "prompt_length_max": 1043,
        "prompt_length_std": 174.62,
        "target_length_mean": 194.71
      },
      {
        "name": "en-fa_ir",
        "sample_count": 960,
        "prompt_length_mean": 261.26,
        "prompt_length_min": 73,
        "prompt_length_max": 1037,
        "prompt_length_std": 174.62,
        "target_length_mean": 181.71
      },
      {
        "name": "en-fi_fi",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 192.93
      },
      {
        "name": "en-fil_ph",
        "sample_count": 960,
        "prompt_length_mean": 267.26,
        "prompt_length_min": 79,
        "prompt_length_max": 1043,
        "prompt_length_std": 174.62,
        "target_length_mean": 222.8
      },
      {
        "name": "en-fr_ca",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 226.12
      },
      {
        "name": "en-fr_fr",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 221.44
      },
      {
        "name": "en-gu_in",
        "sample_count": 960,
        "prompt_length_mean": 267.26,
        "prompt_length_min": 79,
        "prompt_length_max": 1043,
        "prompt_length_std": 174.62,
        "target_length_mean": 192.95
      },
      {
        "name": "en-he_il",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 147.19
      },
      {
        "name": "en-hi_in",
        "sample_count": 960,
        "prompt_length_mean": 261.26,
        "prompt_length_min": 73,
        "prompt_length_max": 1037,
        "prompt_length_std": 174.62,
        "target_length_mean": 204.93
      },
      {
        "name": "en-hr_hr",
        "sample_count": 960,
        "prompt_length_mean": 267.26,
        "prompt_length_min": 79,
        "prompt_length_max": 1043,
        "prompt_length_std": 174.62,
        "target_length_mean": 192.91
      },
      {
        "name": "en-hu_hu",
        "sample_count": 960,
        "prompt_length_mean": 269.26,
        "prompt_length_min": 81,
        "prompt_length_max": 1045,
        "prompt_length_std": 174.62,
        "target_length_mean": 204.03
      },
      {
        "name": "en-id_id",
        "sample_count": 960,
        "prompt_length_mean": 271.26,
        "prompt_length_min": 83,
        "prompt_length_max": 1047,
        "prompt_length_std": 174.62,
        "target_length_mean": 209.26
      },
      {
        "name": "en-is_is",
        "sample_count": 960,
        "prompt_length_mean": 269.26,
        "prompt_length_min": 81,
        "prompt_length_max": 1045,
        "prompt_length_std": 174.62,
        "target_length_mean": 216.09
      },
      {
        "name": "en-it_it",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 209.64
      },
      {
        "name": "en-ja_jp",
        "sample_count": 960,
        "prompt_length_mean": 267.26,
        "prompt_length_min": 79,
        "prompt_length_max": 1043,
        "prompt_length_std": 174.62,
        "target_length_mean": 85.02
      },
      {
        "name": "en-kn_in",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 206.63
      },
      {
        "name": "en-ko_kr",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 100.29
      },
      {
        "name": "en-lt_lt",
        "sample_count": 960,
        "prompt_length_mean": 271.26,
        "prompt_length_min": 83,
        "prompt_length_max": 1047,
        "prompt_length_std": 174.62,
        "target_length_mean": 192.01
      },
      {
        "name": "en-lv_lv",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 188.78
      },
      {
        "name": "en-ml_in",
        "sample_count": 960,
        "prompt_length_mean": 269.26,
        "prompt_length_min": 81,
        "prompt_length_max": 1045,
        "prompt_length_std": 174.62,
        "target_length_mean": 225.76
      },
      {
        "name": "en-mr_in",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 197.82
      },
      {
        "name": "en-nl_nl",
        "sample_count": 960,
        "prompt_length_mean": 261.26,
        "prompt_length_min": 73,
        "prompt_length_max": 1037,
        "prompt_length_std": 174.62,
        "target_length_mean": 215.59
      },
      {
        "name": "en-no_no",
        "sample_count": 960,
        "prompt_length_mean": 269.26,
        "prompt_length_min": 81,
        "prompt_length_max": 1045,
        "prompt_length_std": 174.62,
        "target_length_mean": 189.75
      },
      {
        "name": "en-pa_in",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 201.52
      },
      {
        "name": "en-pl_pl",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 219.99
      },
      {
        "name": "en-pt_br",
        "sample_count": 960,
        "prompt_length_mean": 271.26,
        "prompt_length_min": 83,
        "prompt_length_max": 1047,
        "prompt_length_std": 174.62,
        "target_length_mean": 203.8
      },
      {
        "name": "en-pt_pt",
        "sample_count": 960,
        "prompt_length_mean": 271.26,
        "prompt_length_min": 83,
        "prompt_length_max": 1047,
        "prompt_length_std": 174.62,
        "target_length_mean": 208.8
      },
      {
        "name": "en-ro_ro",
        "sample_count": 960,
        "prompt_length_mean": 267.26,
        "prompt_length_min": 79,
        "prompt_length_max": 1043,
        "prompt_length_std": 174.62,
        "target_length_mean": 208.31
      },
      {
        "name": "en-ru_ru",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 193.64
      },
      {
        "name": "en-sk_sk",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 196.16
      },
      {
        "name": "en-sl_si",
        "sample_count": 960,
        "prompt_length_mean": 269.26,
        "prompt_length_min": 81,
        "prompt_length_max": 1045,
        "prompt_length_std": 174.62,
        "target_length_mean": 194.98
      },
      {
        "name": "en-sr_rs",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 184.37
      },
      {
        "name": "en-sv_se",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 199.59
      },
      {
        "name": "en-sw_ke",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 207.39
      },
      {
        "name": "en-sw_tz",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 200.35
      },
      {
        "name": "en-ta_in",
        "sample_count": 960,
        "prompt_length_mean": 261.26,
        "prompt_length_min": 73,
        "prompt_length_max": 1037,
        "prompt_length_std": 174.62,
        "target_length_mean": 242.31
      },
      {
        "name": "en-te_in",
        "sample_count": 960,
        "prompt_length_mean": 263.26,
        "prompt_length_min": 75,
        "prompt_length_max": 1039,
        "prompt_length_std": 174.62,
        "target_length_mean": 207.96
      },
      {
        "name": "en-th_th",
        "sample_count": 960,
        "prompt_length_mean": 259.26,
        "prompt_length_min": 71,
        "prompt_length_max": 1035,
        "prompt_length_std": 174.62,
        "target_length_mean": 176.53
      },
      {
        "name": "en-tr_tr",
        "sample_count": 960,
        "prompt_length_mean": 265.26,
        "prompt_length_min": 77,
        "prompt_length_max": 1041,
        "prompt_length_std": 174.62,
        "target_length_mean": 195.8
      },
      {
        "name": "en-uk_ua",
        "sample_count": 960,
        "prompt_length_mean": 269.26,
        "prompt_length_min": 81,
        "prompt_length_max": 1045,
        "prompt_length_std": 174.62,
        "target_length_mean": 196.7
      },
      {
        "name": "en-ur_pk",
        "sample_count": 960,
        "prompt_length_mean": 259.26,
        "prompt_length_min": 71,
        "prompt_length_max": 1035,
        "prompt_length_std": 174.62,
        "target_length_mean": 200.79
      },
      {
        "name": "en-vi_vn",
        "sample_count": 960,
        "prompt_length_mean": 271.26,
        "prompt_length_min": 83,
        "prompt_length_max": 1047,
        "prompt_length_std": 174.62,
        "target_length_mean": 206.38
      },
      {
        "name": "en-zh_cn",
        "sample_count": 960,
        "prompt_length_mean": 267.26,
        "prompt_length_min": 79,
        "prompt_length_max": 1043,
        "prompt_length_std": 174.62,
        "target_length_mean": 65.58
      },
      {
        "name": "en-zh_tw",
        "sample_count": 960,
        "prompt_length_mean": 267.26,
        "prompt_length_min": 79,
        "prompt_length_max": 1043,
        "prompt_length_std": 174.62,
        "target_length_mean": 65.97
      },
      {
        "name": "en-zu_za",
        "sample_count": 960,
        "prompt_length_mean": 259.26,
        "prompt_length_min": 71,
        "prompt_length_max": 1035,
        "prompt_length_std": 174.62,
        "target_length_mean": 204.44
      }
    ],
    "prompt_length": {
      "mean": 265.45,
      "min": 71,
      "max": 1047,
      "std": 174.56
    },
    "target_length_mean": 191.7,
    "computed_at": "2026-01-28T15:04:45.147728"
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "557f3aa1",
          "content": [
            {
              "text": "Translate the following english sentence into arabic:\n\nenglish: Siso's depictions of land, water center new gallery exhibition\narabic:"
            }
          ]
        }
      ],
      "target": "رسومات سيسو عن الأرض والمية في معرضه الجديد",
      "id": 0,
      "group_id": 0,
      "subset_key": "en-ar_eg",
      "metadata": {
        "source_text": "Siso's depictions of land, water center new gallery exhibition",
        "target_text": "رسومات سيسو عن الأرض والمية في معرضه الجديد",
        "source_language": "en",
        "target_language": "ar_eg"
      }
    },
    "subset": "en-ar_eg",
    "truncated": false
  },
  "readme": {
    "en": "# WMT2024++\n\n\n## Overview\n\nWMT2024++ is a comprehensive machine translation benchmark based on the WMT 2024 news translation task. It supports 54 language pairs with English as the source language, enabling evaluation of translation quality across diverse target languages.\n\n## Task Description\n\n- **Task Type**: Machine Translation\n- **Input**: Source text in English with translation prompt\n- **Output**: Translated text in the target language\n- **Language Pairs**: 54 pairs (English to 54 target languages)\n\n## Key Features\n\n- Extensive multilingual coverage (54 target languages)\n- News domain text for real-world applicability\n- Multiple evaluation metrics (BLEU, BERTScore, COMET)\n- Standardized prompt template for consistent evaluation\n- Supports batch scoring for efficiency\n\n## Evaluation Notes\n\n- Default configuration uses **0-shot** evaluation\n- Metrics: **BLEU**, **BERTScore** (XLM-RoBERTa), **COMET** (wmt22-comet-da)\n- Evaluates on **test** split\n- Language-specific normalization applied\n- COMET metric requires `unbabel-comet` package\n- Subsets represent individual language pairs (e.g., `en-zh_cn`, `en-de_de`)\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `wmt24pp` |\n| **Dataset ID** | [extraordinarylab/wmt24pp](https://modelscope.cn/datasets/extraordinarylab/wmt24pp/summary) |\n| **Paper** | N/A |\n| **Tags** | `MachineTranslation`, `MultiLingual` |\n| **Metrics** | `bleu`, `bert_score`, `comet` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `test` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 52,800 |\n| Prompt Length (Mean) | 265.45 chars |\n| Prompt Length (Min/Max) | 71 / 1047 chars |\n\n**Per-Subset Statistics:**\n\n| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |\n|--------|---------|-------------|------------|------------|\n| `en-ar_eg` | 960 | 263.26 | 75 | 1039 |\n| `en-ar_sa` | 960 | 263.26 | 75 | 1039 |\n| `en-bg_bg` | 960 | 269.26 | 81 | 1045 |\n| `en-bn_in` | 960 | 265.26 | 77 | 1041 |\n| `en-ca_es` | 960 | 265.26 | 77 | 1041 |\n| `en-cs_cz` | 960 | 261.26 | 73 | 1037 |\n| `en-da_dk` | 960 | 263.26 | 75 | 1039 |\n| `en-de_de` | 960 | 263.26 | 75 | 1039 |\n| `en-el_gr` | 960 | 261.26 | 73 | 1037 |\n| `en-es_mx` | 960 | 265.26 | 77 | 1041 |\n| `en-et_ee` | 960 | 267.26 | 79 | 1043 |\n| `en-fa_ir` | 960 | 261.26 | 73 | 1037 |\n| `en-fi_fi` | 960 | 265.26 | 77 | 1041 |\n| `en-fil_ph` | 960 | 267.26 | 79 | 1043 |\n| `en-fr_ca` | 960 | 263.26 | 75 | 1039 |\n| `en-fr_fr` | 960 | 263.26 | 75 | 1039 |\n| `en-gu_in` | 960 | 267.26 | 79 | 1043 |\n| `en-he_il` | 960 | 263.26 | 75 | 1039 |\n| `en-hi_in` | 960 | 261.26 | 73 | 1037 |\n| `en-hr_hr` | 960 | 267.26 | 79 | 1043 |\n| `en-hu_hu` | 960 | 269.26 | 81 | 1045 |\n| `en-id_id` | 960 | 271.26 | 83 | 1047 |\n| `en-is_is` | 960 | 269.26 | 81 | 1045 |\n| `en-it_it` | 960 | 265.26 | 77 | 1041 |\n| `en-ja_jp` | 960 | 267.26 | 79 | 1043 |\n| `en-kn_in` | 960 | 265.26 | 77 | 1041 |\n| `en-ko_kr` | 960 | 263.26 | 75 | 1039 |\n| `en-lt_lt` | 960 | 271.26 | 83 | 1047 |\n| `en-lv_lv` | 960 | 265.26 | 77 | 1041 |\n| `en-ml_in` | 960 | 269.26 | 81 | 1045 |\n| `en-mr_in` | 960 | 265.26 | 77 | 1041 |\n| `en-nl_nl` | 960 | 261.26 | 73 | 1037 |\n| `en-no_no` | 960 | 269.26 | 81 | 1045 |\n| `en-pa_in` | 960 | 265.26 | 77 | 1041 |\n| `en-pl_pl` | 960 | 263.26 | 75 | 1039 |\n| `en-pt_br` | 960 | 271.26 | 83 | 1047 |\n| `en-pt_pt` | 960 | 271.26 | 83 | 1047 |\n| `en-ro_ro` | 960 | 267.26 | 79 | 1043 |\n| `en-ru_ru` | 960 | 265.26 | 77 | 1041 |\n| `en-sk_sk` | 960 | 263.26 | 75 | 1039 |\n| `en-sl_si` | 960 | 269.26 | 81 | 1045 |\n| `en-sr_rs` | 960 | 265.26 | 77 | 1041 |\n| `en-sv_se` | 960 | 265.26 | 77 | 1041 |\n| `en-sw_ke` | 960 | 265.26 | 77 | 1041 |\n| `en-sw_tz` | 960 | 265.26 | 77 | 1041 |\n| `en-ta_in` | 960 | 261.26 | 73 | 1037 |\n| `en-te_in` | 960 | 263.26 | 75 | 1039 |\n| `en-th_th` | 960 | 259.26 | 71 | 1035 |\n| `en-tr_tr` | 960 | 265.26 | 77 | 1041 |\n| `en-uk_ua` | 960 | 269.26 | 81 | 1045 |\n| `en-ur_pk` | 960 | 259.26 | 71 | 1035 |\n| `en-vi_vn` | 960 | 271.26 | 83 | 1047 |\n| `en-zh_cn` | 960 | 267.26 | 79 | 1043 |\n| `en-zh_tw` | 960 | 267.26 | 79 | 1043 |\n| `en-zu_za` | 960 | 259.26 | 71 | 1035 |\n\n## Sample Example\n\n**Subset**: `en-ar_eg`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"557f3aa1\",\n      \"content\": [\n        {\n          \"text\": \"Translate the following english sentence into arabic:\\n\\nenglish: Siso's depictions of land, water center new gallery exhibition\\narabic:\"\n        }\n      ]\n    }\n  ],\n  \"target\": \"رسومات سيسو عن الأرض والمية في معرضه الجديد\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"subset_key\": \"en-ar_eg\",\n  \"metadata\": {\n    \"source_text\": \"Siso's depictions of land, water center new gallery exhibition\",\n    \"target_text\": \"رسومات سيسو عن الأرض والمية في معرضه الجديد\",\n    \"source_language\": \"en\",\n    \"target_language\": \"ar_eg\"\n  }\n}\n```\n\n## Prompt Template\n\n**Prompt Template:**\n```text\nTranslate the following {source_language} sentence into {target_language}:\n\n{source_language}: {source_text}\n{target_language}:\n```\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets wmt24pp \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['wmt24pp'],\n    dataset_args={\n        'wmt24pp': {\n            # subset_list: ['en-ar_eg', 'en-ar_sa', 'en-bg_bg']  # optional, evaluate specific subsets\n        }\n    },\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# WMT2024++\n\n\n## 概述\n\nWMT2024++ 是一个基于 WMT 2024 新闻翻译任务的综合性机器翻译基准测试。它支持以英语为源语言的 54 个语言对，可用于评估模型在多种目标语言上的翻译质量。\n\n## 任务描述\n\n- **任务类型**：机器翻译\n- **输入**：带有翻译提示的英文源文本\n- **输出**：目标语言的翻译文本\n- **语言对**：54 个（英语到 54 种目标语言）\n\n## 主要特性\n\n- 广泛的多语言覆盖（54 种目标语言）\n- 新闻领域文本，贴近实际应用场景\n- 多种评估指标（BLEU、BERTScore、COMET）\n- 标准化的提示模板，确保评估一致性\n- 支持批量评分以提升效率\n\n## 评估说明\n\n- 默认配置使用 **0-shot** 评估\n- 评估指标：**BLEU**、**BERTScore**（XLM-RoBERTa）、**COMET**（wmt22-comet-da）\n- 在 **test** 划分上进行评估\n- 应用语言特定的归一化处理\n- COMET 指标需要安装 `unbabel-comet` 包\n- 子集代表单个语言对（例如 `en-zh_cn`、`en-de_de`）\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `wmt24pp` |\n| **数据集ID** | [extraordinarylab/wmt24pp](https://modelscope.cn/datasets/extraordinarylab/wmt24pp/summary) |\n| **论文** | N/A |\n| **标签** | `MachineTranslation`, `MultiLingual` |\n| **指标** | `bleu`, `bert_score`, `comet` |\n| **默认示例数** | 0-shot |\n| **评估划分** | `test` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 52,800 |\n| 提示词长度（平均） | 265.45 字符 |\n| 提示词长度（最小/最大） | 71 / 1047 字符 |\n\n**各子集统计数据：**\n\n| 子集 | 样本数 | 提示平均长度 | 提示最小长度 | 提示最大长度 |\n|--------|---------|-------------|------------|------------|\n| `en-ar_eg` | 960 | 263.26 | 75 | 1039 |\n| `en-ar_sa` | 960 | 263.26 | 75 | 1039 |\n| `en-bg_bg` | 960 | 269.26 | 81 | 1045 |\n| `en-bn_in` | 960 | 265.26 | 77 | 1041 |\n| `en-ca_es` | 960 | 265.26 | 77 | 1041 |\n| `en-cs_cz` | 960 | 261.26 | 73 | 1037 |\n| `en-da_dk` | 960 | 263.26 | 75 | 1039 |\n| `en-de_de` | 960 | 263.26 | 75 | 1039 |\n| `en-el_gr` | 960 | 261.26 | 73 | 1037 |\n| `en-es_mx` | 960 | 265.26 | 77 | 1041 |\n| `en-et_ee` | 960 | 267.26 | 79 | 1043 |\n| `en-fa_ir` | 960 | 261.26 | 73 | 1037 |\n| `en-fi_fi` | 960 | 265.26 | 77 | 1041 |\n| `en-fil_ph` | 960 | 267.26 | 79 | 1043 |\n| `en-fr_ca` | 960 | 263.26 | 75 | 1039 |\n| `en-fr_fr` | 960 | 263.26 | 75 | 1039 |\n| `en-gu_in` | 960 | 267.26 | 79 | 1043 |\n| `en-he_il` | 960 | 263.26 | 75 | 1039 |\n| `en-hi_in` | 960 | 261.26 | 73 | 1037 |\n| `en-hr_hr` | 960 | 267.26 | 79 | 1043 |\n| `en-hu_hu` | 960 | 269.26 | 81 | 1045 |\n| `en-id_id` | 960 | 271.26 | 83 | 1047 |\n| `en-is_is` | 960 | 269.26 | 81 | 1045 |\n| `en-it_it` | 960 | 265.26 | 77 | 1041 |\n| `en-ja_jp` | 960 | 267.26 | 79 | 1043 |\n| `en-kn_in` | 960 | 265.26 | 77 | 1041 |\n| `en-ko_kr` | 960 | 263.26 | 75 | 1039 |\n| `en-lt_lt` | 960 | 271.26 | 83 | 1047 |\n| `en-lv_lv` | 960 | 265.26 | 77 | 1041 |\n| `en-ml_in` | 960 | 269.26 | 81 | 1045 |\n| `en-mr_in` | 960 | 265.26 | 77 | 1041 |\n| `en-nl_nl` | 960 | 261.26 | 73 | 1037 |\n| `en-no_no` | 960 | 269.26 | 81 | 1045 |\n| `en-pa_in` | 960 | 265.26 | 77 | 1041 |\n| `en-pl_pl` | 960 | 263.26 | 75 | 1039 |\n| `en-pt_br` | 960 | 271.26 | 83 | 1047 |\n| `en-pt_pt` | 960 | 271.26 | 83 | 1047 |\n| `en-ro_ro` | 960 | 267.26 | 79 | 1043 |\n| `en-ru_ru` | 960 | 265.26 | 77 | 1041 |\n| `en-sk_sk` | 960 | 263.26 | 75 | 1039 |\n| `en-sl_si` | 960 | 269.26 | 81 | 1045 |\n| `en-sr_rs` | 960 | 265.26 | 77 | 1041 |\n| `en-sv_se` | 960 | 265.26 | 77 | 1041 |\n| `en-sw_ke` | 960 | 265.26 | 77 | 1041 |\n| `en-sw_tz` | 960 | 265.26 | 77 | 1041 |\n| `en-ta_in` | 960 | 261.26 | 73 | 1037 |\n| `en-te_in` | 960 | 263.26 | 75 | 1039 |\n| `en-th_th` | 960 | 259.26 | 71 | 1035 |\n| `en-tr_tr` | 960 | 265.26 | 77 | 1041 |\n| `en-uk_ua` | 960 | 269.26 | 81 | 1045 |\n| `en-ur_pk` | 960 | 259.26 | 71 | 1035 |\n| `en-vi_vn` | 960 | 271.26 | 83 | 1047 |\n| `en-zh_cn` | 960 | 267.26 | 79 | 1043 |\n| `en-zh_tw` | 960 | 267.26 | 79 | 1043 |\n| `en-zu_za` | 960 | 259.26 | 71 | 1035 |\n\n## 样例示例\n\n**子集**: `en-ar_eg`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"557f3aa1\",\n      \"content\": [\n        {\n          \"text\": \"Translate the following english sentence into arabic:\\n\\nenglish: Siso's depictions of land, water center new gallery exhibition\\narabic:\"\n        }\n      ]\n    }\n  ],\n  \"target\": \"رسومات سيسو عن الأرض والمية في معرضه الجديد\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"subset_key\": \"en-ar_eg\",\n  \"metadata\": {\n    \"source_text\": \"Siso's depictions of land, water center new gallery exhibition\",\n    \"target_text\": \"رسومات سيسو عن الأرض والمية في معرضه الجديد\",\n    \"source_language\": \"en\",\n    \"target_language\": \"ar_eg\"\n  }\n}\n```\n\n## 提示模板\n\n**提示模板：**\n```text\nTranslate the following {source_language} sentence into {target_language}:\n\n{source_language}: {source_text}\n{target_language}:\n```\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets wmt24pp \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['wmt24pp'],\n    dataset_args={\n        'wmt24pp': {\n            # subset_list: ['en-ar_eg', 'en-ar_sa', 'en-bg_bg']  # 可选，用于评估特定子集\n        }\n    },\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "f0c96a413a0c2e0d95e52561f2c11a7c",
    "needs_translation": false
  },
  "updated_at": "2026-01-28T17:31:32.679612",
  "translation_updated_at": "2026-01-28T16:09:53Z"
}