{"id":1688,"date":"2026-06-05T17:42:27","date_gmt":"2026-06-05T09:42:27","guid":{"rendered":"https:\/\/www.izhuhn.cn\/?p=1688"},"modified":"2026-07-13T16:38:35","modified_gmt":"2026-07-13T08:38:35","slug":"evalscope-%e7%a6%bb%e7%ba%bf%e8%af%84%e6%b5%8b%e9%83%a8%e7%bd%b2%e7%ae%80%e8%ae%b0%ef%bc%88%e5%8d%8e%e4%b8%ba910b-openai-%e5%85%bc%e5%ae%b9%e6%8e%a5%e5%8f%a3%ef%bc%89","status":"publish","type":"post","link":"https:\/\/www.izhuhn.cn\/index.php\/2026\/06\/05\/evalscope-%e7%a6%bb%e7%ba%bf%e8%af%84%e6%b5%8b%e9%83%a8%e7%bd%b2%e7%ae%80%e8%ae%b0%ef%bc%88%e5%8d%8e%e4%b8%ba910b-openai-%e5%85%bc%e5%ae%b9%e6%8e%a5%e5%8f%a3%ef%bc%89\/","title":{"rendered":"EvalScope \u79bb\u7ebf\u8bc4\u6d4b\u90e8\u7f72\u7b80\u8bb0\uff08\u534e\u4e3a910B ARM + OpenAI \u517c\u5bb9\u63a5\u53e3\uff09"},"content":{"rendered":"\n

EvalScope \u79bb\u7ebf\u8bc4\u6d4b\u90e8\u7f72\u7b80\u8bb0\uff08\u534e\u4e3a910B ARM + OpenAI \u517c\u5bb9\u63a5\u53e3\uff09<\/h1>\n

\u80cc\u666f<\/h2>\n

\u5728\u534e\u4e3a 910B (ARM64) \u65ad\u7f51\u670d\u52a1\u5668\u4e0a\u8dd1 EvalScope \u8bc4\u6d4b\u6846\u67b6\u3002Intel Mac \u4e0a\u4ea4\u53c9\u51c6\u5907 ARM64 \u955c\u50cf\u548c\u6570\u636e\u96c6\uff0c\u6253\u5305\u540e scp \u5bfc\u5165\u5373\u53ef\u3002\u8986\u76d6\u901a\u7528\u77e5\u8bc6\u3001\u6570\u5b66\u63a8\u7406\u3001\u4ee3\u7801\u3001Agent\/\u5de5\u5177\u8c03\u7528\u56db\u4e2a\u7ef4\u5ea6\uff0c\u516b\u4e2a\u6807\u51c6\u6570\u636e\u96c6\u3002<\/p>\n

\u8bc4\u6d4b\u6846\u67b6\uff1aEvalScope<\/a> v1.7.1\uff0cModelScope \u793e\u533a\u7ef4\u62a4\u3002<\/p>\n

\u91cd\u8981\u66f4\u65b0\uff082026-06-10\uff09\uff1a\u5b9e\u9645\u90e8\u7f72\u4e2d\u53d1\u73b0\u82e5\u5e72\u914d\u7f6e\u9677\u9631\uff0c\u5df2\u4fee\u6b63\u5e76\u8865\u5145\u811a\u672c\u5316\u65b9\u6848\uff0c\u89c1\u7b2c\u516b\u8282\u8e29\u5751\u8bb0\u5f55\u3002\u7ecf\u5b9e\u9645\u9a8c\u8bc1\uff0cBFCL-v4 \u6570\u636e\u786e\u5b9e\u5185\u7f6e\u5728 bfcl-eval \u5305\u4e2d\uff0812MB\uff09\uff0c\u65e0\u9700\u989d\u5916\u4e0b\u8f7d\u3002<\/strong><\/p>\n

\u601d\u8def<\/h2>\n

\u6574\u4e2a\u79bb\u7ebf\u65b9\u6848\u8981\u89e3\u51b3\u4e09\u4e2a\u6838\u5fc3\u77db\u76fe\uff1a<\/p>\n

    \n
  1. \u65ad\u7f51 vs \u6570\u636e\u96c6\u4e0b\u8f7d<\/strong>\uff1a\u6570\u636e\u96c6\u5728 ModelScope\/HuggingFace \u4e0a\uff0cNPU \u8282\u70b9\u65e0\u5916\u7f51\u3002\u601d\u8def\u662f\u5728 Mac \u4e0a\u9884\u5148\u4e0b\u8f7d\uff0c\u6253\u5305\u540e scp \u4f20\u8f93\uff0c\u901a\u8fc7 Volume \u6302\u8f7d\u8fdb\u5bb9\u5668\u3002<\/li>\n
  2. ARM64 vs x86 Mac<\/strong>\uff1a910B \u662f ARM64\uff0cMac \u662f Intel x86\u3002\u7528 Docker Buildx \u4ea4\u53c9\u7f16\u8bd1 ARM64 \u955c\u50cf\uff0c\u8fd9\u6837\u5bb9\u5668\u5185\u7684 Python \u73af\u5883\uff08Python 3.10 + ARM64 \u4f9d\u8d56\uff09\u548c NPU \u539f\u751f\u73af\u5883\u4e00\u81f4\u3002<\/li>\n
  3. \u591a\u79cd\u6570\u636e\u96c6 vs \u7edf\u4e00\u5165\u53e3<\/strong>\uff1a\u516b\u4e2a\u6570\u636e\u96c6\u5206\u5c5e\u56db\u4e2a\u7ef4\u5ea6\uff0c\u6bcf\u4e2a\u53c2\u6570\u4e0d\u540c\uff08few-shot \u6570\u91cf\u3001\u8d85\u65f6\u3001\u5b50\u96c6\u9009\u62e9\uff09\u3002\u5982\u679c\u5168\u5199\u6b7b\u4e00\u4e2a\u547d\u4ee4\uff0c\u8981\u4e48\u592a\u5197\u4f59\u8981\u4e48\u8dd1\u4e0d\u5168\u3002\u65b9\u6848\u662f\u7528\u4e24\u4e2a\u811a\u672c + \u73af\u5883\u53d8\u91cf\u9a71\u52a8\uff0c\u6309\u7ef4\u5ea6\u5206\u7c7b\u8c03\u7528\u3002<\/li>\n<\/ol>\n

    \u4e3a\u4ec0\u4e48\u955c\u50cf\u548c\u6570\u636e\u5206\u5f00\uff1f<\/h3>\n

    \u955c\u50cf\u7ea6 10GB\uff0c\u6570\u636e\u96c6\u7ea6 700MB\u3002\u5982\u679c\u5408\u5728\u4e00\u8d77\uff0c\u6bcf\u6b21\u66f4\u65b0\u6570\u636e\u96c6\u90fd\u9700\u8981\u91cd\u65b0\u6784\u5efa\u6574\u4e2a\u955c\u50cf\u2014\u2014\u8017\u65f6\u4e14\u6d6a\u8d39\u4f20\u8f93\u5e26\u5bbd\u3002\u5206\u5f00\u540e\uff0c\u955c\u50cf\u53ea\u9700\u6784\u5efa\u4e00\u6b21\uff0c\u6570\u636e\u96c6\u72ec\u7acb\u66f4\u65b0\u3001\u6302\u8f7d\u5373\u53ef\u3002<\/p>\n

    \u4e3a\u4ec0\u4e48\u4e0d\u7528 EvalScope \u81ea\u5e26\u7684 dataset-hub\uff1f<\/h3>\n

    EvalScope \u9ed8\u8ba4\u901a\u8fc7 ModelScope Hub \u81ea\u52a8\u4e0b\u8f7d\u6570\u636e\u96c6\u5230 --dataset-dir<\/code> \u6307\u5b9a\u76ee\u5f55\u3002\u8fd9\u5728\u65ad\u7f51\u73af\u5883\u6709\u4e24\u4e2a\u95ee\u9898\uff1a<\/p>\n

      \n
    • \u627e\u4e0d\u5230\u7f13\u5b58\u65f6\u4f1a\u5c1d\u8bd5\u8054\u7f51\u4e0b\u8f7d \u2192 DNS \u8d85\u65f6\u62a5\u9519<\/li>\n
    • \u7f13\u5b58\u76ee\u5f55\u7ed3\u6784\uff08{org}\/{repo}\/<\/code>\uff09\u548c EvalScope \u7684\u9ed8\u8ba4\u67e5\u627e\u8def\u5f84\uff08~\/.cache\/modelscope\/hub\/datasets\/<\/code>\uff09\u4e0d\u4e00\u81f4\uff0c\u5bb9\u6613\u51fa\u73b0\u8def\u5f84\u5339\u914d\u5931\u8d25<\/li>\n<\/ul>\n

      \u6240\u4ee5\u76f4\u63a5\u7528 dataset_id<\/code> \u53c2\u6570\u6307\u5b9a\u672c\u5730\u7edd\u5bf9\u8def\u5f84\uff0c\u5b8c\u5168\u7ed5\u8fc7 Hub \u673a\u5236\uff0c\u79bb\u7ebf\u65e0\u4f9d\u8d56\u3002<\/p>\n

      \u4e3a\u4ec0\u4e48\u5206\u4e24\u4e2a\u811a\u672c\uff1f<\/h3>\n

      run_eval.sh<\/code> \u5904\u7406\u6807\u51c6\u6570\u636e\u96c6\uff08mmlu\u3001ceval\u3001humaneval \u7b49\uff09\uff0c\u8fd9\u4e9b\u6570\u636e\u96c6\u9700\u8981\u6302\u8f7d modelscope_cache<\/code> \u76ee\u5f55\uff0c\u6709 few_shot_num<\/code>\u3001subset_list<\/code> \u7b49\u53c2\u6570\u3002<\/p>\n

      run_agent_eval.sh<\/code> \u5904\u7406 Agent \u6570\u636e\u96c6\uff08bfcl_v4\u3001tau_bench\uff09\uff0c\u8fd9\u4e9b\u6570\u636e\u96c6\u6570\u636e\u5185\u7f6e\u5728 Python \u5305\u4e2d\uff0c\u4e0d\u9700\u8981\u989d\u5916\u6302\u8f7d\uff0c\u4f46\u9700\u8981\u7279\u6b8a\u7684 extra_params<\/code>\uff08\u5982 tau_bench \u7684\u7528\u6237\u6a21\u62df\u6a21\u578b\u914d\u7f6e\uff09\u3002<\/p>\n

      \u4e24\u7c7b\u6570\u636e\u96c6\u7684\u6302\u8f7d\u9700\u6c42\u3001\u53c2\u6570\u7ed3\u6784\u5b8c\u5168\u4e0d\u540c\uff0c\u5206\u5f00\u6bd4\u6df7\u5728\u4e00\u8d77\u66f4\u6e05\u6670\u3002<\/p>\n

      \u4e00\u3001Mac \u7aef\u51c6\u5907\uff08\u6784\u5efa\u955c\u50cf\u4e0e\u4e0b\u8f7d\u6570\u636e\u96c6\uff09<\/h2>\n

      Intel Mac \u4e0a\u901a\u8fc7 Docker Buildx \u4ea4\u53c9\u7f16\u8bd1 ARM64 \u955c\u50cf\u3002\u6838\u5fc3\u4f9d\u8d56\u662f QEMU \u591a\u67b6\u6784\u6a21\u62df\u5c42\u3002<\/p>\n

      1.1 \u524d\u7f6e\u6761\u4ef6<\/h3>\n
      <\/span># Docker Desktop 4.47+\uff08\u652f\u6301 macOS 13.7 + buildx + QEMU\uff09<\/span>\ndocker <\/span>--version        <\/span># Docker version 28.x+<\/span>\ndocker <\/span>buildx <\/span>version   <\/span># \u5fc5\u987b\u652f\u6301 --platform linux\/arm64<\/span>\n\n# \u6ce8\u518c QEMU \u591a\u67b6\u6784\u652f\u6301\uff08\u53ea\u9700\u8981\u6267\u884c\u4e00\u6b21\uff0c\u91cd\u542f\u540e\u9700\u91cd\u65b0\u6267\u884c\uff09<\/span>\ndocker <\/span>run <\/span>--rm <\/span>--privileged <\/span>multiarch\/qemu-user-static <\/span>--reset <\/span>-p <\/span>yes\n\n# \u9a8c\u8bc1 ARM64 \u6784\u5efa\u53ef\u7528<\/span>\ndocker <\/span>buildx <\/span>inspect <\/span>--bootstrap\n# \u5e94\u770b\u5230 Platforms: linux\/arm64, linux\/amd64 \u7b49<\/span>\n<\/code><\/pre><\/div>\n\n
      1.2 Dockerfile \u7ed3\u6784<\/h3>\n
      <\/span>FROM<\/span> <\/span>python:3.10-slim-bookworm<\/span>          <\/span># ARM64 Python \u57fa\u7840\u955c\u50cf<\/span>\n\n# \u73af\u5883\u53d8\u91cf\u9884\u8bbe\u79bb\u7ebf\u548c\u7f13\u5b58\u8def\u5f84<\/span>\nENV<\/span> <\/span>PYTHONUNBUFFERED<\/span>=<\/span>1<\/span> <\/span>\\<\/span>\n    <\/span>EVALSCOPE_CACHE<\/span>=<\/span>\/workspace\/cache\/evalscope <\/span>\\<\/span>\n    <\/span>MODELSCOPE_CACHE<\/span>=<\/span>\/workspace\/cache\/modelscope\/hub <\/span>\\<\/span>\n    <\/span>HF_HOME<\/span>=<\/span>\/workspace\/cache\/huggingface <\/span>\\<\/span>\n    <\/span>HF_HUB_OFFLINE<\/span>=<\/span>1<\/span> <\/span>\\<\/span>\n    <\/span>TRANSFORMERS_OFFLINE<\/span>=<\/span>1<\/span>\n\n# \u5b89\u88c5\u6784\u5efa\u4f9d\u8d56 + docker.io\uff08SWE-bench \u9700\u8981\uff09+ git\/git-lfs<\/span>\nRUN<\/span> <\/span>apt-get <\/span>update <\/span>&&<\/span> <\/span>apt-get <\/span>install <\/span>-y <\/span>--no-install-recommends <\/span>\\<\/span>\n        <\/span>build-essential <\/span>ca-certificates <\/span>curl <\/span>docker.io <\/span>\\<\/span>\n        <\/span>git <\/span>git-lfs <\/span>jq <\/span>openssh-client <\/span>patch <\/span>tini <\/span>\\<\/span>\n    <\/span>&&<\/span> <\/span>rm <\/span>-rf <\/span>\/var\/lib\/apt\/lists\/*\n\n# \u5b89\u88c5 Python \u4f9d\u8d56\uff08evalscope + bfcl-eval + swebench + tau_bench\uff09<\/span>\nCOPY<\/span> <\/span>requirements.txt <\/span>\/workspace\/requirements.txt\nRUN<\/span> <\/span>pip <\/span>install <\/span>--upgrade <\/span>pip <\/span>&&<\/span> <\/span>pip <\/span>install <\/span>-r <\/span>\/workspace\/requirements.txt\n\n# \u62f7\u5165\u8bc4\u6d4b\u811a\u672c\u548c\u914d\u7f6e<\/span>\nCOPY<\/span> <\/span>scripts <\/span>\/workspace\/scripts\nCOPY<\/span> <\/span>configs <\/span>\/workspace\/configs\n\nENTRYPOINT<\/span> <\/span>[<\/span>"\/usr\/bin\/tini"<\/span>,<\/span> <\/span>"--"<\/span>]<\/span>\nCMD<\/span> <\/span>[<\/span>"bash"<\/span>]<\/span>\n<\/code><\/pre><\/div>\n\n
      \u51e0\u4e2a\u8bbe\u8ba1\u8003\u91cf\uff1a\n- \u9009 Python 3.10-slim<\/strong> \u800c\u975e 3.12\/3.13\uff1aEvalScope 1.7.1 \u7684\u4f9d\u8d56\uff08\u7279\u522b\u662f bfcl-eval \u5bf9 faiss-cpu==1.11.0<\/code> \u7684\u786c\u7f16\u7801\u8981\u6c42\uff09\u5728 Python 3.12+ \u4e0a\u5b58\u5728\u517c\u5bb9\u95ee\u9898\uff0c3.10 \u662f\u6700\u7a33\u59a5\u7684\u9009\u62e9\u3002\n- \u5b89\u88c5 docker.io<\/strong>\uff1aSWE-bench \u8bc4\u6d4b\u9700\u8981\u5728\u5bb9\u5668\u5185\u8c03\u7528 Docker \u6765\u6784\u5efa\u548c\u8fd0\u884c\u6d4b\u8bd5\u73af\u5883\uff0c\u56e0\u6b64\u955c\u50cf\u5185\u9700\u8981 Docker CLI\uff08\u901a\u8fc7 -v \/var\/run\/docker.sock<\/code> \u4e0e\u5bbf\u4e3b\u673a Docker daemon \u901a\u4fe1\uff09\u3002\n- HF_HUB_OFFLINE=1<\/code><\/strong>\uff1a\u5f3a\u5236 HuggingFace \u5e93\u8fdb\u5165\u79bb\u7ebf\u6a21\u5f0f\uff0c\u5373\u4f7f\u4ee3\u7801\u4e2d\u6709\u8054\u7f51\u5c1d\u8bd5\u4e5f\u4f1a\u76f4\u63a5\u4f7f\u7528\u672c\u5730\u7f13\u5b58\uff0c\u907f\u514d\u610f\u5916\u4e0b\u8f7d\u3002\n- tini \u4f5c\u4e3a ENTRYPOINT<\/strong>\uff1a\u8f7b\u91cf\u7ea7 init \u8fdb\u7a0b\uff0c\u5904\u7406\u4fe1\u53f7\u8f6c\u53d1\u548c\u50f5\u5c38\u8fdb\u7a0b\u56de\u6536\uff0c\u5bf9\u957f\u65f6\u95f4\u8fd0\u884c\u7684\u8bc4\u6d4b\u4efb\u52a1\u6bd4\u8f83\u53cb\u597d\u3002<\/p>\n
      1.3 \u4f9d\u8d56\u8bf4\u660e<\/h3>\n
      <\/span>evalscope[swe-bench,bfcl]==1.7.1    # EvalScope \u6838\u5fc3 + SWE-bench + BFCL \u63d2\u4ef6\nmodelscope>=1.31.0                    # ModelScope SDK\uff08\u4e0b\u8f7d\u6570\u636e\u96c6\u7528\uff09\ndatasets>=3.0.0                       # HuggingFace datasets \u5e93\nbfcl-eval==2025.10.27.1              # BFCL-v4 \u8bc4\u6d4b\u5f15\u64ce\uff08\u6570\u636e\u5185\u7f6e\uff09\nswebench==4.1.0                       # SWE-bench \u8bc4\u6d4b\u5f15\u64ce\ngit+https:\/\/github.com\/sierra-research\/tau-bench  # tau_bench \u7528\u6237\u6a21\u62df\n<\/code><\/pre><\/div>\n\n
      evalscope[swe-bench,bfcl]<\/code> \u662f EvalScope \u7684 extras \u5b89\u88c5\u65b9\u5f0f\u2014\u2014swe-bench<\/code> \u9644\u52a0 SWE-bench \u9002\u914d\u5668\uff0cbfcl<\/code> \u9644\u52a0 BFCL \u9002\u914d\u5668\u3002<\/p>\n
      1.4 \u4e00\u952e\u6784\u5efa\u811a\u672c<\/h3>\n
      <\/span>cd<\/span> <\/span>evalscope_offline\nbash <\/span>scripts\/download_assets.sh\n<\/code><\/pre><\/div>\n\n
      \u8fd9\u4e2a\u811a\u672c\u6309\u987a\u5e8f\u6267\u884c\uff1a<\/p>\n
        \n
      1. \u4e0b\u8f7d Python wheels \u5230 wheels\/<\/code><\/strong>\uff08\u53ef\u9009\uff0cMac ARM64 \u4ea4\u53c9\u4e0b\u8f7d\u53ef\u80fd\u5931\u8d25\uff0c\u6b64\u65f6\u8df3\u8fc7\u4e0d\u5f71\u54cd\u540e\u7eed\uff09<\/li>\n
      2. \u4e0b\u8f7d ModelScope \u6570\u636e\u96c6\u5230 modelscope_cache\/<\/code><\/strong>\uff1a\u8c03\u7528 ModelScope snapshot_download<\/code> \u4e0b\u8f7d\u5168\u90e8 7 \u4e2a\u6570\u636e\u96c6\uff0c\u7ea6 5-10 \u5206\u949f\uff08\u53d6\u51b3\u4e8e\u7f51\u7edc\uff09<\/li>\n
      3. \u6784\u5efa ARM64 Docker \u955c\u50cf<\/strong>\uff1adocker buildx build --platform linux\/arm64 -t evalscope-offline:arm64 . --load<\/code>\uff0c\u7ea6 15-30 \u5206\u949f\uff08\u53d6\u51b3\u4e8e\u7f51\u7edc\u548c CPU\uff09<\/li>\n
      4. \u5bfc\u51fa\u955c\u50cf\u4e3a tar.gz<\/strong>\uff1adocker save | gzip<\/code>\uff0c\u4ea7\u7269\u7ea6 6GB<\/li>\n<\/ol>\n
        \u5982\u679c\u6b65\u9aa4 1 \u5931\u8d25\uff0c\u53ef\u4ee5\u8df3\u8fc7\uff1a<\/p>\n
        <\/span># \u624b\u52a8\u5206\u6b65\u6267\u884c<\/span>\npython3 <\/span>scripts\/download_datasets.py <\/span>--cache-dir <\/span>modelscope_cache\n\ndocker <\/span>buildx <\/span>build <\/span>--platform <\/span>linux\/arm64 <\/span>\\<\/span>\n  <\/span>-t <\/span>evalscope-offline:arm64 <\/span>\\<\/span>\n  <\/span>-f <\/span>Dockerfile.arm64 <\/span>. <\/span>--load\n\ndocker <\/span>save <\/span>evalscope-offline:arm64 <\/span>|<\/span> <\/span>gzip <\/span>-c <\/span>> <\/span>images\/evalscope-offline-arm64.tar.gz\n<\/code><\/pre><\/div>\n\n
        1.5 \u76ee\u5f55\u7ed3\u6784\u4e0e\u4ea7\u7269<\/h3>\n
        \u6784\u5efa\u5b8c\u6210\u540e Mac \u4e0a\u7684\u76ee\u5f55\uff1a<\/p>\n
        <\/span>evalscope_offline\/\n\u251c\u2500\u2500 Dockerfile.arm64                    # ARM64 \u955c\u50cf\u5b9a\u4e49\n\u251c\u2500\u2500 requirements.txt                    # Python \u4f9d\u8d56\n\u251c\u2500\u2500 modelscope_cache\/                   # \u6570\u636e\u96c6 (709MB) \u2014 \u9700\u8981 scp \u5230 NPU\n\u2502   \u251c\u2500\u2500 cais\/mmlu\/\n\u2502   \u251c\u2500\u2500 evalscope\/ceval\/\n\u2502   \u251c\u2500\u2500 AI-ModelScope\/gpqa_diamond\/\n\u2502   \u251c\u2500\u2500 ZhipuAI\/LongBench-v2\/\n\u2502   \u251c\u2500\u2500 knoveleng\/Minerva-Math\/\n\u2502   \u251c\u2500\u2500 opencompass\/humaneval\/\n\u2502   \u2514\u2500\u2500 evalscope\/swe-bench-verified-mini\/\n\u251c\u2500\u2500 images\/\n\u2502   \u2514\u2500\u2500 evalscope-offline-arm64.tar.gz  # Docker \u955c\u50cf (6GB) \u2014 \u9700\u8981 scp \u5230 NPU\n\u251c\u2500\u2500 scripts\/                            # \u8bc4\u6d4b\u811a\u672c \u2014 scp \u5230 NPU\n\u2502   \u251c\u2500\u2500 run_eval.sh\n\u2502   \u251c\u2500\u2500 run_agent_eval.sh\n\u2502   \u2514\u2500\u2500 run_smoke_mock_api.py\n\u251c\u2500\u2500 configs\/                            # \u53c2\u8003\u914d\u7f6e\n\u2502   \u251c\u2500\u2500 910b_eval_core.yaml\n\u2502   \u251c\u2500\u2500 910b_eval_agent.yaml\n\u2502   \u2514\u2500\u2500 910b_eval_swe.yaml\n\u2514\u2500\u2500 wheels\/                             # Python \u79bb\u7ebf\u5b89\u88c5\u5305\uff08\u53ef\u9009\uff09\n<\/code><\/pre><\/div>\n\n
        \u4e8c\u3001\u4ea7\u7269\u8bf4\u660e\u4e0e\u4f20\u8f93<\/h2>\n
        \u4e09\u4e2a\u6587\u4ef6\u9700\u8981\u4f20\u5230 NPU\uff1a<\/p>\n
        <\/span>1.<\/span> <\/span>images<\/span>\/<\/span>evalscope<\/span>-<\/span>offline<\/span>-<\/span>arm64<\/span>.<\/span>tar<\/span>.<\/span>gz<\/span>  <\/span>(<\/span>6.2<\/span>GB<\/span>)<\/span>   <\/span>Docker<\/span> <\/span>\u955c\u50cf<\/span>\n2.<\/span> <\/span>modelscope_cache<\/span>\/<\/span>                      <\/span>(<\/span>709<\/span>MB<\/span>)<\/span>   <\/span>\u6570\u636e\u96c6\u7f13\u5b58<\/span>\n3.<\/span> <\/span>scripts<\/span>\/<\/span>                                         <\/span>\u8bc4\u6d4b\u811a\u672c<\/span>\n<\/code><\/pre><\/div>\n\n
        \u5176\u4e2d\uff0cBFCL-v4 \u548c tau_bench \u7684\u8bc4\u6d4b\u6570\u636e\u5206\u522b\u5185\u7f6e\u5728 bfcl-eval<\/code> pip \u5305\u548c tau-bench<\/code> git \u5305\u4e2d\uff0cDocker \u955c\u50cf\u5df2\u5305\u542b\uff0c\u65e0\u9700\u989d\u5916\u4f20\u8f93\u3002<\/p>\n
        2.1 \u6587\u4ef6\u4f20\u8f93<\/h3>\n
        <\/span># \u4ece Mac \u4f20\u5230 910B<\/span>\nscp <\/span>images\/evalscope-offline-arm64.tar.gz <\/span>user@910b:\/data\/eval\/\nscp <\/span>-r <\/span>modelscope_cache\/ <\/span>user@910b:\/data\/eval\/\nscp <\/span>-r <\/span>scripts\/ <\/span>user@910b:\/data\/eval\/\n<\/code><\/pre><\/div>\n\n
        \u4e09\u3001\u8986\u76d6\u7ef4\u5ea6\u4e0e\u6570\u636e\u96c6<\/h2>\n\n\n\n\n\n\n\n\n\n\n\n\n\n
        \u7ef4\u5ea6<\/th>\n\u6570\u636e\u96c6<\/th>\n\u6837\u672c\u91cf<\/th>\n\u8bc4\u6d4b\u6307\u6807<\/th>\nfew-shot<\/th>\n<\/tr>\n<\/thead>\n
        \u901a\u7528\u77e5\u8bc6<\/td>\nMMLU<\/td>\n~14k<\/td>\nAccuracy<\/td>\n5<\/td>\n<\/tr>\n
        \u901a\u7528\u77e5\u8bc6<\/td>\nC-Eval<\/td>\n~14k<\/td>\nAccuracy<\/td>\n5<\/td>\n<\/tr>\n
        \u901a\u7528\u77e5\u8bc6<\/td>\nGPQA Diamond<\/td>\n198<\/td>\nAccuracy<\/td>\n0<\/td>\n<\/tr>\n
        \u6570\u5b66\u63a8\u7406<\/td>\nMinerva-Math<\/td>\n272<\/td>\nAccuracy<\/td>\n0<\/td>\n<\/tr>\n
        \u957f\u6587\u672c\u63a8\u7406<\/td>\nLongBench-v2<\/td>\n100<\/td>\nAccuracy<\/td>\n-<\/td>\n<\/tr>\n
        \u4ee3\u7801<\/td>\nHumanEval<\/td>\n164<\/td>\npass@1<\/td>\n0<\/td>\n<\/tr>\n
        \u4ee3\u7801<\/td>\nSWE-bench-mini<\/td>\n50<\/td>\nresolve rate<\/td>\n0<\/td>\n<\/tr>\n
        Agent\/\u5de5\u5177\u8c03\u7528<\/td>\nBFCL-v4<\/td>\n~1700<\/td>\nAccuracy<\/td>\n0<\/td>\n<\/tr>\n
        Agent<\/td>\ntau_bench<\/td>\n\u591a\u57df<\/td>\nTask Success<\/td>\n0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
        \u56db\u3001910B \u79bb\u7ebf\u90e8\u7f72<\/h2>\n
        4.1 \u52a0\u8f7d\u955c\u50cf<\/h3>\n
        <\/span>ssh <\/span>user@910b\ncd<\/span> <\/span>\/data\/eval\ndocker <\/span>load <\/span>-i <\/span>evalscope-offline-arm64.tar.gz\n# \u9a8c\u8bc1<\/span>\ndocker <\/span>images <\/span>evalscope-offline:arm64\n# \u8f93\u51fa: evalscope-offline:arm64 ... ~10GB<\/span>\n<\/code><\/pre><\/div>\n\n
        4.2 \u76ee\u5f55\u89c4\u5212<\/h3>\n
        <\/span>mkdir <\/span>-p <\/span>\/data\/eval\/reports <\/span>\/data\/eval\/outputs\n\n# \u6700\u7ec8\u7ed3\u6784<\/span>\n\/data\/eval\/\n\u251c\u2500\u2500 <\/span>modelscope_cache\/      <\/span># \u6838\u5fc3\u6570\u636e\u96c6\uff08\u4e0d\u52a0 :ro\uff09<\/span>\n\u251c\u2500\u2500 <\/span>reports\/               <\/span># \u8bc4\u6d4b\u62a5\u544a\u8f93\u51fa<\/span>\n\u251c\u2500\u2500 <\/span>outputs\/               <\/span># \u4e2d\u95f4\u7ed3\u679c<\/span>\n\u2514\u2500\u2500 <\/span>scripts\/               <\/span># \u8bc4\u6d4b\u811a\u672c<\/span>\n<\/code><\/pre><\/div>\n\n
        \u4e3a\u4ec0\u4e48\u8981 --network host<\/code>\uff1f<\/strong> \u5bb9\u5668\u9700\u8981\u8bbf\u95ee\u5bbf\u4e3b\u673a\u7684\u6a21\u578b\u63a8\u7406\u670d\u52a1\uff08910B NPU \u4e0a\u8fd0\u884c\u7684 API\uff09\u3002\u5982\u679c\u7528\u9ed8\u8ba4\u6865\u63a5\u7f51\u7edc\uff0c\u5bb9\u5668\u5185\u8bbf\u95ee\u5bbf\u4e3b IP \u9700\u8981\u989d\u5916\u914d\u7f6e\u3002--network host<\/code> \u6700\u7b80\u5355\u53ef\u9760\u3002<\/p>\n
        \u4e3a\u4ec0\u4e48 modelscope_cache \u4e0d\u52a0 :ro<\/code>\uff1f<\/strong> EvalScope \u8fd0\u884c\u65f6\u4f1a\u5f80\u6570\u636e\u96c6\u76ee\u5f55\u5199\u5165\u7f13\u5b58\u7d22\u5f15\uff08Arrow\/Parquet \u683c\u5f0f\u9996\u6b21\u52a0\u8f7d\u65f6\u6846\u67b6\u4f1a\u751f\u6210\u7d22\u5f15\u6587\u4ef6\uff09\u3002\u53ea\u8bfb\u6302\u8f7d\u4f1a\u5bfc\u81f4 Permission denied<\/code>\uff0c\u8bc4\u6d4b\u76f4\u63a5\u62a5\u9519\u3002\u867d\u7136\u6570\u636e\u96c6\u672c\u8eab\u4e0d\u9700\u8981\u88ab\u4fee\u6539\uff0c\u4f46\u6846\u67b6\u9700\u8981\u5199\u6743\u9650\u5efa\u7acb\u7d22\u5f15\u3002\u5982\u679c\u786e\u5b9e\u4e0d\u60f3\u8ba9\u6570\u636e\u96c6\u76ee\u5f55\u88ab\u4fee\u6539\uff0c\u53ef\u4ee5 cp -r<\/code> \u4e00\u4efd\u526f\u672c\u518d\u6302\u8f7d\u3002<\/p>\n
        4.3 \u524d\u63d0\uff1a\u6a21\u578b\u63a8\u7406\u670d\u52a1<\/h3>\n
        910B \u4e0a\u5fc5\u987b\u8fd0\u884c\u4e00\u4e2a OpenAI \u517c\u5bb9\u7684 \/v1<\/code> API<\/strong>\uff1a<\/p>\n
        <\/span>curl <\/span>http:\/\/127.0.0.1:8000\/v1\/models\n# \u9884\u671f\u8fd4\u56de: {"object":"list","data":[{"id":"\u4f60\u7684\u6a21\u578b\u540d","object":"model"}]}<\/span>\n<\/code><\/pre><\/div>\n\n
        EvalScope \u7684 openai_api<\/code> \u6a21\u5f0f\u672c\u8d28\u4e0a\u662f\u4e00\u4e2a HTTP \u5ba2\u6237\u7aef\uff0c\u5411\u6307\u5b9a URL \u53d1\u9001 Chat Completions \u683c\u5f0f\u8bf7\u6c42\u3002\u5b83\u5bf9\u6a21\u578b\u670d\u52a1\u7684\u8981\u6c42\u76f8\u5bf9\u5bbd\u677e\uff1a\u8fd4\u56de\u7684 JSON \u9aa8\u67b6\uff08id<\/code>\u3001object<\/code>\u3001choices<\/code>\uff09\u5b8c\u6574\u5373\u53ef\uff0ctoken \u6570\u586b 0 \u4e0d\u5f71\u54cd\u8bc4\u6d4b\u3002<\/p>\n
        \u5982\u679c\u4f60\u7684 910B \u6a21\u578b\u670d\u52a1\u4e0d\u662f OpenAI \u683c\u5f0f\uff0c\u53c2\u8003\u7b2c\u516d\u8282\u9002\u914d\u5c42\u65b9\u6848\u3002<\/p>\n
        4.4 \u8fd0\u884c\u8bc4\u6d4b<\/h3>\n
        \u4e24\u4e2a\u811a\u672c\u5bf9\u5e94\u4e24\u7c7b\u8bc4\u6d4b\u573a\u666f\uff1a<\/p>\n\n\n\n\n\n\n
        \u811a\u672c<\/th>\n\u7528\u9014<\/th>\n\u6570\u636e\u96c6<\/th>\n<\/tr>\n<\/thead>\n
        run_eval.sh<\/code><\/td>\n\u77e5\u8bc6\/\u63a8\u7406\/\u4ee3\u7801<\/td>\nmmlu, ceval, gpqa_diamond, minerva_math, longbench_v2, humaneval, swe_bench_verified_mini<\/td>\n<\/tr>\n
        run_agent_eval.sh<\/code><\/td>\nAgent\/\u5de5\u5177\u8c03\u7528<\/td>\nbfcl_v4, tau_bench<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
        \u4e00\u3001\u77e5\u8bc6\u7c7b\uff08Knowledge\uff09<\/h4>\n
        <\/span>docker <\/span>run <\/span>--rm <\/span>--network <\/span>host <\/span>\\<\/span>\n  <\/span>-e <\/span>MODEL<\/span>=<\/span>qwen3-next <\/span>\\<\/span>\n  <\/span>-e <\/span>API_URL<\/span>=<\/span>http:\/\/10.7.6.39:3000\/v1 <\/span>\\<\/span>\n  <\/span>-e <\/span>DATASETS<\/span>=<\/span>"mmlu ceval gpqa_diamond"<\/span> <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/modelscope_cache:\/workspace\/cache\/modelscope <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/reports:\/workspace\/reports <\/span>\\<\/span>\n  <\/span>evalscope-offline:arm64 <\/span>\\<\/span>\n  <\/span>bash <\/span>\/workspace\/scripts\/run_eval.sh\n<\/code><\/pre><\/div>\n\n
        \u4e8c\u3001\u63a8\u7406\u7c7b\uff08Math\/reasoning\uff09<\/h4>\n
        <\/span>docker <\/span>run <\/span>--rm <\/span>--network <\/span>host <\/span>\\<\/span>\n  <\/span>-e <\/span>MODEL<\/span>=<\/span>qwen3-next <\/span>\\<\/span>\n  <\/span>-e <\/span>API_URL<\/span>=<\/span>http:\/\/10.7.6.39:3000\/v1 <\/span>\\<\/span>\n  <\/span>-e <\/span>DATASETS<\/span>=<\/span>"minerva_math longbench_v2"<\/span> <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/modelscope_cache:\/workspace\/cache\/modelscope <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/reports:\/workspace\/reports <\/span>\\<\/span>\n  <\/span>evalscope-offline:arm64 <\/span>\\<\/span>\n  <\/span>bash <\/span>\/workspace\/scripts\/run_eval.sh\n<\/code><\/pre><\/div>\n\n
        \u4e09\u3001\u4ee3\u7801\u7c7b\uff08Coding\uff09<\/h4>\n
        <\/span># HumanEval\uff08\u57fa\u7840\u4ee3\u7801\u751f\u6210\uff09<\/span>\ndocker <\/span>run <\/span>--rm <\/span>--network <\/span>host <\/span>\\<\/span>\n  <\/span>-e <\/span>MODEL<\/span>=<\/span>qwen3-next <\/span>\\<\/span>\n  <\/span>-e <\/span>API_URL<\/span>=<\/span>http:\/\/10.7.6.39:3000\/v1 <\/span>\\<\/span>\n  <\/span>-e <\/span>DATASETS<\/span>=<\/span>humaneval <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/modelscope_cache:\/workspace\/cache\/modelscope <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/reports:\/workspace\/reports <\/span>\\<\/span>\n  <\/span>evalscope-offline:arm64 <\/span>\\<\/span>\n  <\/span>bash <\/span>\/workspace\/scripts\/run_eval.sh\n\n# SWE-bench\uff08\u9700\u8981 Docker socket\uff09<\/span>\ndocker <\/span>run <\/span>--rm <\/span>--network <\/span>host <\/span>\\<\/span>\n  <\/span>-e <\/span>MODEL<\/span>=<\/span>qwen3-next <\/span>\\<\/span>\n  <\/span>-e <\/span>API_URL<\/span>=<\/span>http:\/\/10.7.6.39:3000\/v1 <\/span>\\<\/span>\n  <\/span>-e <\/span>DATASETS<\/span>=<\/span>swe_bench_verified_mini <\/span>\\<\/span>\n  <\/span>-e <\/span>TIMEOUT<\/span>=<\/span>300<\/span> <\/span>\\<\/span>\n  <\/span>-e <\/span>MAX_TOKENS<\/span>=<\/span>4096<\/span> <\/span>\\<\/span>\n  <\/span>-v <\/span>\/var\/run\/docker.sock:\/var\/run\/docker.sock <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/modelscope_cache:\/workspace\/cache\/modelscope <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/reports:\/workspace\/reports <\/span>\\<\/span>\n  <\/span>evalscope-offline:arm64 <\/span>\\<\/span>\n  <\/span>bash <\/span>\/workspace\/scripts\/run_eval.sh\n<\/code><\/pre><\/div>\n\n
        SWE-bench \u6bd4\u8f83\u7279\u6b8a\uff0c\u5b83\u4e0d\u4ec5\u8bc4\u6d4b\u4ee3\u7801\u751f\u6210\uff0c\u8fd8\u8981\u5728 Docker \u73af\u5883\u4e2d\u5b9e\u9645\u8fd0\u884c\u6d4b\u8bd5\u7528\u4f8b\u6765\u9a8c\u8bc1\u4ee3\u7801\u4fee\u590d\u662f\u5426\u6b63\u786e\u2014\u2014\u6240\u4ee5\u5fc5\u987b\u6302\u8f7d \/var\/run\/docker.sock<\/code>\uff0c\u8ba9\u5bb9\u5668\u5185\u7684 Docker CLI \u8fde\u63a5\u5bbf\u4e3b\u673a daemon \u542f\u52a8\u6d4b\u8bd5\u5bb9\u5668\u3002TIMEOUT=300<\/code> \u662f\u56e0\u4e3a\u62c9\u53d6\/\u6784\u5efa Docker \u955c\u50cf\u548c\u8fd0\u884c\u6d4b\u8bd5\u6bd4\u8f83\u8017\u65f6\u3002<\/p>\n
        \u56db\u3001Agent\/\u5de5\u5177\u8c03\u7528<\/h4>\n
        <\/span># BFCL-v4\uff08\u6570\u636e\u5185\u7f6e\u5728\u955c\u50cf\u4e2d\uff0c\u65e0\u9700\u989d\u5916\u6302\u8f7d\uff09<\/span>\ndocker <\/span>run <\/span>--rm <\/span>--network <\/span>host <\/span>\\<\/span>\n  <\/span>-e <\/span>MODEL<\/span>=<\/span>qwen3-next <\/span>\\<\/span>\n  <\/span>-e <\/span>API_URL<\/span>=<\/span>http:\/\/10.7.6.39:3000\/v1 <\/span>\\<\/span>\n  <\/span>-e <\/span>DATASETS<\/span>=<\/span>bfcl_v4 <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/reports:\/workspace\/reports <\/span>\\<\/span>\n  <\/span>evalscope-offline:arm64 <\/span>\\<\/span>\n  <\/span>bash <\/span>\/workspace\/scripts\/run_agent_eval.sh\n\n# tau_bench\uff08\u9700\u8981\u7528\u6237\u6a21\u62df\u6a21\u578b\uff09<\/span>\ndocker <\/span>run <\/span>--rm <\/span>--network <\/span>host <\/span>\\<\/span>\n  <\/span>-e <\/span>MODEL<\/span>=<\/span>qwen3-next <\/span>\\<\/span>\n  <\/span>-e <\/span>API_URL<\/span>=<\/span>http:\/\/10.7.6.39:3000\/v1 <\/span>\\<\/span>\n  <\/span>-e <\/span>DATASETS<\/span>=<\/span>tau_bench <\/span>\\<\/span>\n  <\/span>-e <\/span>TAU_USER_MODEL<\/span>=<\/span>qwen3-next <\/span>\\<\/span>\n  <\/span>-e <\/span>TAU_USER_API_BASE<\/span>=<\/span>http:\/\/10.7.6.39:3000\/v1 <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/reports:\/workspace\/reports <\/span>\\<\/span>\n  <\/span>evalscope-offline:arm64 <\/span>\\<\/span>\n  <\/span>bash <\/span>\/workspace\/scripts\/run_agent_eval.sh\n<\/code><\/pre><\/div>\n\n
        tau_bench \u7684\u5de5\u4f5c\u539f\u7406\u662f\u7528\u4e00\u4e2a\"\u7528\u6237\u6a21\u62df\u6a21\u578b\"\u626e\u6f14\u5ba2\u6237\u89d2\u8272\uff0c\u4e0e\u88ab\u6d4b\u6a21\u578b\u8fdb\u884c\u591a\u8f6e\u5bf9\u8bdd\u3002\u8fd9\u91cc TAU_USER_MODEL<\/code> \u590d\u7528\u4e86\u88ab\u6d4b\u6a21\u578b\u672c\u8eab\u2014\u2014\u5982\u679c\u6761\u4ef6\u5141\u8bb8\uff0c\u7528\u53e6\u4e00\u4e2a\u72ec\u7acb\u7684\u8f83\u5c0f\u6a21\u578b\u6548\u679c\u66f4\u597d\uff08\u6a21\u62df\u7528\u6237\u4e0d\u9700\u8981\u5f88\u5f3a\uff09\u3002<\/p>\n
        \u6d4b\u8bd5\u7528\u5c0f\u6837\u672c\uff1a<\/strong> -e LIMIT=10<\/code>\uff08\u52a0\u5728\u4ee5\u4e0a\u4efb\u610f\u547d\u4ee4\u4e2d\uff0c\u6b63\u5f0f\u8dd1\u65f6\u53bb\u6389\u5373\u8dd1\u5168\u91cf\uff09\u3002<\/p>\n
        BFCL subset \u8bf4\u660e\uff1a<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
        \u5b50\u96c6<\/th>\n\u542b\u4e49<\/th>\n\u8bf4\u660e<\/th>\n<\/tr>\n<\/thead>\n
        simple_python\/java\/javascript<\/code><\/td>\n\u5355\u51fd\u6570\u8c03\u7528<\/td>\n\u57fa\u7840<\/td>\n<\/tr>\n
        multiple<\/code><\/td>\n\u591a\u51fd\u6570\u4e32\u884c\u8c03\u7528<\/td>\n\u4e2d\u7b49<\/td>\n<\/tr>\n
        parallel<\/code><\/td>\n\u5e76\u884c\u591a\u51fd\u6570\u8c03\u7528<\/td>\n\u4e2d\u7b49<\/td>\n<\/tr>\n
        parallel_multiple<\/code><\/td>\n\u5e76\u884c+\u4e32\u884c\u6df7\u7528<\/td>\n\u8f83\u96be<\/td>\n<\/tr>\n
        irrelevance<\/code><\/td>\n\u65e0\u5173\u51fd\u6570\u5e72\u6270<\/td>\n\u8003\u5bdf\u5224\u65ad\u529b<\/td>\n<\/tr>\n
        multi_turn_*<\/code><\/td>\n\u591a\u8f6e\u5bf9\u8bdd<\/td>\nAgent \u80fd\u529b<\/td>\n<\/tr>\n
        live_*<\/code><\/td>\n\u9700\u8981\u771f\u5b9e API \u8c03\u7528<\/td>\n\u79bb\u7ebf\u8df3\u8fc7<\/td>\n<\/tr>\n
        memory_*<\/code><\/td>\n\u9700\u8981\u5411\u91cf\u8bb0\u5fc6<\/td>\n\u9700\u8981 sentence-transformers\uff0c\u9ed8\u8ba4\u955c\u50cf\u672a\u5b89\u88c5<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
        4.5 \u76f4\u63a5 CLI \u65b9\u5f0f\uff08\u4e0d\u7528\u811a\u672c\uff09<\/h3>\n
        \u811a\u672c\u65b9\u5f0f\u7b49\u4ef7\u4e8e\u4ee5\u4e0b\u539f\u751f\u547d\u4ee4\uff0c\u65b9\u4fbf\u7406\u89e3\u53c2\u6570\u542b\u4e49\uff1a<\/p>\n
        <\/span>docker <\/span>run <\/span>--rm <\/span>--network <\/span>host <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/modelscope_cache:\/workspace\/cache\/modelscope <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/reports:\/workspace\/reports <\/span>\\<\/span>\n  <\/span>evalscope-offline:arm64 <\/span>\\<\/span>\n  <\/span>evalscope <\/span>eval<\/span> <\/span>\\<\/span>\n    <\/span>--model <\/span>qwen3-next <\/span>\\<\/span>\n    <\/span>--api-url <\/span>http:\/\/10.7.6.39:3000\/v1 <\/span>\\<\/span>\n    <\/span>--api-key <\/span>***"***\\" \\<\/span>\n    --eval-type openai_api \\<\/span>\n    --datasets mmlu ceval \\<\/span>\n    --limit 10 \\<\/span>\n    --dataset-args '{\\"mmlu\\": {\\"dataset_id\\": \\"\/workspace\/cache\/modelscope\/cais\/mmlu\\", \\"few_shot_num\\": 5}, \\"ceval\\": {\\"dataset_id\\": \\"\/workspace\/cache\/modelscope\/evalscope\/ceval\\", \\"few_shot_num\\": 5}}' \\<\/span>\n    --generation-config '{\\"temperature\\":0,\\"max_tokens\\":2048,\\"timeout\\":180}'\\n```\\n\\n**\u6ce8\u610f\uff1a** `dataset_id` \u53c2\u6570\u76f4\u63a5\u6307\u5b9a\u672c\u5730\u8def\u5f84\uff08`local_path` \u5df2\u5e9f\u5f03\uff09\uff0c\u4e0d\u9700\u8981 `--dataset-hub` \u548c `--dataset-dir`\u3002<\/span>\n\n## \u4e94\u3001OpenAI \u6807\u51c6\u63a5\u53e3\u683c\u5f0f\u8981\u6c42<\/span>\n\nEvalScope `openai_api` \u6a21\u5f0f\u5bf9\u6a21\u578b\u670d\u52a1\u7684\u8981\u6c42\u5982\u4e0b\u3002<\/span>\n\n### 5.1 GET \/v1\/models<\/span>\n\n```json<\/span>\n{<\/span>\n  "<\/span>object": "<\/span>list",<\/span>\n  "<\/span>data": [{"<\/span>id": "<\/span>your-model-name", "<\/span>object": "<\/span>model"}]<\/span>\n}<\/span>\n<\/code><\/pre><\/div>\n\n
        5.2 POST \/v1\/chat\/completions<\/h3>\n
        \u6b63\u5e38\u8fd4\u56de<\/strong>\uff1a<\/p>\n
        <\/span>{<\/span>\n  <\/span>"id"<\/span>:<\/span> <\/span>"chatcmpl-xxx"<\/span>,<\/span>\n  <\/span>"object"<\/span>:<\/span> <\/span>"chat.completion"<\/span>,<\/span>\n  <\/span>"model"<\/span>:<\/span> <\/span>"your-model-name"<\/span>,<\/span>\n  <\/span>"choices"<\/span>:<\/span> <\/span>[{<\/span>\n    <\/span>"index"<\/span>:<\/span> <\/span>0<\/span>,<\/span>\n    <\/span>"message"<\/span>:<\/span> <\/span>{<\/span>"role"<\/span>:<\/span> <\/span>"assistant"<\/span>,<\/span> <\/span>"content"<\/span>:<\/span> <\/span>"\u56de\u7b54\u5185\u5bb9"<\/span>},<\/span>\n    <\/span>"finish_reason"<\/span>:<\/span> <\/span>"stop"<\/span>\n  <\/span>}],<\/span>\n  <\/span>"usage"<\/span>:<\/span> <\/span>{<\/span>"prompt_tokens"<\/span>:<\/span> <\/span>100<\/span>,<\/span> <\/span>"completion_tokens"<\/span>:<\/span> <\/span>50<\/span>,<\/span> <\/span>"total_tokens"<\/span>:<\/span> <\/span>150<\/span>}<\/span>\n}<\/span>\n<\/code><\/pre><\/div>\n\n
        \u5de5\u5177\u8c03\u7528\u8fd4\u56de<\/strong>\uff08BFCL \u573a\u666f\uff0ccontent \u4e3a null\uff09\uff1a<\/p>\n
        <\/span>{<\/span>\n  <\/span>"choices"<\/span>:<\/span> <\/span>[{<\/span>\n    <\/span>"message"<\/span>:<\/span> <\/span>{<\/span>\n      <\/span>"role"<\/span>:<\/span> <\/span>"assistant"<\/span>,<\/span>\n      <\/span>"content"<\/span>:<\/span> <\/span>null<\/span>,<\/span>\n      <\/span>"tool_calls"<\/span>:<\/span> <\/span>[{<\/span>\n        <\/span>"id"<\/span>:<\/span> <\/span>"call_abc123"<\/span>,<\/span>\n        <\/span>"type"<\/span>:<\/span> <\/span>"function"<\/span>,<\/span>\n        <\/span>"function"<\/span>:<\/span> <\/span>{<\/span>"name"<\/span>:<\/span> <\/span>"get_weather"<\/span>,<\/span> <\/span>"arguments"<\/span>:<\/span> <\/span>"{\\"city\\": \\"Beijing\\"}"<\/span>}<\/span>\n      <\/span>}]<\/span>\n    <\/span>},<\/span>\n    <\/span>"finish_reason"<\/span>:<\/span> <\/span>"tool_calls"<\/span>\n  <\/span>}]<\/span>\n}<\/span>\n<\/code><\/pre><\/div>\n\n
        usage<\/code> \u5b57\u6bb5\u7684 token \u6570\u586b 0 \u4e0d\u5f71\u54cd\u8bc4\u6d4b\u7ed3\u679c\uff0c\u4f46 id<\/code>\u3001object<\/code>\u3001choices<\/code> \u9aa8\u67b6\u5fc5\u987b\u5b8c\u6574\u3002<\/p>\n
        \u516d\u3001\u975e\u6807\u51c6\u63a5\u53e3\u9002\u914d\u65b9\u6848<\/h2>\n
        \u5982\u679c 910B \u6a21\u578b\u670d\u52a1\u4e0d\u662f OpenAI \u683c\u5f0f\uff0c\u7528 FastAPI \u5199\u8f6c\u6362\u5c42\uff1a<\/p>\n
        <\/span>from<\/span> <\/span>fastapi<\/span> <\/span>import<\/span> FastAPI<\/span>\nimport<\/span> <\/span>httpx<\/span>\nimport<\/span> <\/span>uuid<\/span>\n\napp<\/span> =<\/span> FastAPI<\/span>()<\/span>\nYOUR_ENDPOINT<\/span> =<\/span> "http:\/\/127.0.0.1:9090\/generate"<\/span>\n\n@app<\/span>.<\/span>get<\/span>(<\/span>"\/v1\/models"<\/span>)<\/span>\nasync<\/span> def<\/span> <\/span>models<\/span>():<\/span>\n    return<\/span> {<\/span>"object"<\/span>:<\/span> "list"<\/span>,<\/span> "data"<\/span>:<\/span> [{<\/span>"id"<\/span>:<\/span> "my-model"<\/span>,<\/span> "object"<\/span>:<\/span> "model"<\/span>}]}<\/span>\n\n@app<\/span>.<\/span>post<\/span>(<\/span>"\/v1\/chat\/completions"<\/span>)<\/span>\nasync<\/span> def<\/span> <\/span>chat<\/span>(<\/span>req<\/span>:<\/span> dict<\/span>):<\/span>\n    messages<\/span> =<\/span> req<\/span>.<\/span>get<\/span>(<\/span>"messages"<\/span>,<\/span> [])<\/span>\n    tools<\/span> =<\/span> req<\/span>.<\/span>get<\/span>(<\/span>"tools"<\/span>)<\/span>\n    prompt<\/span> =<\/span> messages<\/span>[<\/span>-<\/span>1<\/span>]<\/span>.<\/span>get<\/span>(<\/span>"content"<\/span>,<\/span> ""<\/span>)<\/span> if<\/span> messages<\/span> else<\/span> ""<\/span>\n    if<\/span> messages<\/span> and<\/span> messages<\/span>[<\/span>0<\/span>]<\/span>.<\/span>get<\/span>(<\/span>"role"<\/span>)<\/span> ==<\/span> "system"<\/span>:<\/span>\n        prompt<\/span> =<\/span> messages<\/span>[<\/span>0<\/span>][<\/span>"content"<\/span>]<\/span> +<\/span> "<\/span>\\n\\n<\/span>"<\/span> +<\/span> prompt<\/span>\n    if<\/span> tools<\/span>:<\/span>\n        tool_desc<\/span> =<\/span> "<\/span>\\n<\/span>"<\/span>.<\/span>join<\/span>(<\/span>\n            f<\/span>"Function: <\/span>{<\/span>t<\/span>[<\/span>'function'<\/span>][<\/span>'name'<\/span>]<\/span>}<\/span>\\n<\/span>{<\/span>t<\/span>[<\/span>'function'<\/span>][<\/span>'description'<\/span>]<\/span>}<\/span>"<\/span>\n            for<\/span> t<\/span> in<\/span> tools<\/span>\n        )<\/span>\n        prompt<\/span> =<\/span> f<\/span>"<\/span>{<\/span>tool_desc<\/span>}<\/span>\\n\\n<\/span>User request: <\/span>{<\/span>prompt<\/span>}<\/span>\\n\\n<\/span>Respond with function call JSON."<\/span>\n    payload<\/span> =<\/span> {<\/span>\n        "prompt"<\/span>:<\/span> prompt<\/span>,<\/span>\n        "max_new_tokens"<\/span>:<\/span> req<\/span>.<\/span>get<\/span>(<\/span>"max_tokens"<\/span>,<\/span> 2048<\/span>),<\/span>\n        "temperature"<\/span>:<\/span> req<\/span>.<\/span>get<\/span>(<\/span>"temperature"<\/span>,<\/span> 0<\/span>),<\/span>\n    }<\/span>\n    async<\/span> with<\/span> httpx<\/span>.<\/span>AsyncClient<\/span>()<\/span> as<\/span> client<\/span>:<\/span>\n        resp<\/span> =<\/span> await<\/span> client<\/span>.<\/span>post<\/span>(<\/span>YOUR_ENDPOINT<\/span>,<\/span> json<\/span>=<\/span>payload<\/span>,<\/span> timeout<\/span>=<\/span>300<\/span>)<\/span>\n    result<\/span> =<\/span> resp<\/span>.<\/span>json<\/span>()<\/span>\n    return<\/span> {<\/span>\n        "id"<\/span>:<\/span> f<\/span>"chatcmpl-<\/span>{<\/span>uuid<\/span>.<\/span>uuid4<\/span>()<\/span>.<\/span>hex<\/span>[:<\/span>12<\/span>]<\/span>}<\/span>"<\/span>,<\/span>\n        "object"<\/span>:<\/span> "chat.completion"<\/span>,<\/span>\n        "model"<\/span>:<\/span> req<\/span>[<\/span>"model"<\/span>],<\/span>\n        "choices"<\/span>:<\/span> [{<\/span>\n            "index"<\/span>:<\/span> 0<\/span>,<\/span>\n            "message"<\/span>:<\/span> {<\/span>"role"<\/span>:<\/span> "assistant"<\/span>,<\/span> "content"<\/span>:<\/span> result<\/span>[<\/span>"text"<\/span>]},<\/span>\n            "finish_reason"<\/span>:<\/span> "stop"<\/span>\n        }],<\/span>\n        "usage"<\/span>:<\/span> {<\/span>"prompt_tokens"<\/span>:<\/span> 0<\/span>,<\/span> "completion_tokens"<\/span>:<\/span> 0<\/span>,<\/span> "total_tokens"<\/span>:<\/span> 0<\/span>}<\/span>\n    }<\/span>\n<\/code><\/pre><\/div>\n\n
        \u542f\u52a8\uff1a<\/p>\n
        <\/span>pip <\/span>install <\/span>fastapi <\/span>uvicorn <\/span>httpx\nuvicorn <\/span>adapter:app <\/span>--host <\/span>0<\/span>.0.0.0 <\/span>--port <\/span>8000<\/span>\n<\/code><\/pre><\/div>\n\n
        \u7136\u540e EvalScope \u6307\u5411 http:\/\/127.0.0.1:8000\/v1<\/code>\u3002<\/p>\n
        \u4e03\u3001\u5192\u70df\u9a8c\u8bc1<\/h2>\n
        \u5728\u6b63\u5f0f\u8dd1\u8bc4\u6d4b\u4e4b\u524d\uff0c\u5efa\u8bae\u5148\u7528\u5185\u7f6e\u7684 mock API \u9a8c\u8bc1\u6574\u6761\u94fe\u8def\u662f\u5426\u6253\u901a\u2014\u2014\u955c\u50cf\u662f\u5426\u6b63\u786e\u52a0\u8f7d\u3001\u6570\u636e\u96c6\u6302\u8f7d\u662f\u5426\u751f\u6548\u3001EvalScope \u662f\u5426\u80fd\u6b63\u5e38\u8c03\u5ea6\u3002<\/p>\n
        \u601d\u8def\u5f88\u7b80\u5355\uff1a\u542f\u52a8\u4e00\u4e2a mock API\uff08\u4e0d\u7ba1\u95ee\u4ec0\u4e48\u90fd\u8fd4\u56de\u56fa\u5b9a\u7b54\u6848\uff09\uff0c\u7136\u540e\u8dd1\u4e00\u6761 HumanEval \u6837\u672c\u3002\u5982\u679c\u80fd\u6b63\u5e38\u8dd1\u5b8c\uff08Benchmark finished\uff09\uff0c\u8bf4\u660e\u94fe\u8def\u901a\u4e86\uff1b\u5982\u679c\u5361\u5728\u6570\u636e\u96c6\u52a0\u8f7d\u9636\u6bb5\uff0c\u8bf4\u660e\u6302\u8f7d\u95ee\u9898\u3002<\/p>\n
        <\/span># \u7ec8\u7aef1\uff1a\u542f\u52a8 mock API<\/span>\ndocker <\/span>run <\/span>--rm <\/span>-p <\/span>18080<\/span>:18080 <\/span>evalscope-offline:arm64 <\/span>\\<\/span>\n  <\/span>python <\/span>\/workspace\/scripts\/run_smoke_mock_api.py <\/span>--port <\/span>18080<\/span>\n\n# \u7ec8\u7aef2\uff1a\u5192\u70df\u6d4b\u8bd5<\/span>\ndocker <\/span>run <\/span>--rm <\/span>--network <\/span>host <\/span>\\<\/span>\n  <\/span>-e <\/span>MODEL<\/span>=<\/span>mock <\/span>\\<\/span>\n  <\/span>-e <\/span>API_URL<\/span>=<\/span>http:\/\/127.0.0.1:18080\/v1 <\/span>\\<\/span>\n  <\/span>-e <\/span>DATASETS<\/span>=<\/span>humaneval <\/span>\\<\/span>\n  <\/span>-e <\/span>LIMIT<\/span>=<\/span>1<\/span> <\/span>\\<\/span>\n  <\/span>-v <\/span>\/data\/eval\/modelscope_cache:\/workspace\/cache\/modelscope <\/span>\\<\/span>\n  <\/span>evalscope-offline:arm64 <\/span>\\<\/span>\n  <\/span>bash <\/span>\/workspace\/scripts\/run_eval.sh\n<\/code><\/pre><\/div>\n\n
        \u770b\u5230 Benchmark humaneval evaluation finished<\/code> \u5373\u94fe\u8def\u6b63\u5e38\u3002<\/p>\n
        \u516b\u3001\u8e29\u5751\u8bb0\u5f55<\/h2>\n
        \u5b9e\u9645\u90e8\u7f72\u4e2d\u9047\u5230\u7684\u51e0\u4e2a\u95ee\u9898\u53ca\u89e3\u51b3\u65b9\u6848\u3002<\/p>\n
        8.1 dataset_dir \u8def\u5f84\u9519\u8bef<\/h3>\n
        \u73b0\u8c61<\/strong>\uff1aevalscope \u62a5 Running[eval] 0%| 0\/0<\/code>\uff0c0 \u6761 benchmark\uff0c\u8f93\u51fa DummyCustomModel<\/code>\u3002<\/p>\n
        \u539f\u56e0<\/strong>\uff1a--dataset-dir<\/code> \u8bbe\u7f6e\u4e86\u9519\u8bef\u7684\u8def\u5f84 \/workspace\/cache\/modelscope\/datasets<\/code>\u3002ModelScope \u7684 snapshot_download(cache_dir=cache)<\/code> \u5c06\u6570\u636e\u96c6\u4e0b\u8f7d\u5230 {cache}\/{org}\/{repo}\/<\/code> \u7ed3\u6784\uff0c\u800c\u975e {cache}\/datasets\/{org}\/{repo}\/<\/code>\u3002\u52a0 \/datasets<\/code> \u540e\u7f00\u53cd\u800c\u8ba9\u8def\u5f84\u4e0d\u5bf9\u3002<\/p>\n
        \u4fee\u6b63<\/strong>\uff1a\u76f4\u63a5\u4e0d\u7528 --dataset-hub<\/code> \u548c --dataset-dir<\/code>\uff0c\u6539\u7528 dataset_id<\/code> \u6307\u5b9a\u7edd\u5bf9\u8def\u5f84\u3002<\/p>\n
        8.2 \u4e0d\u8981\u7528 :ro \u53ea\u8bfb\u6302\u8f7d<\/h3>\n
        EvalScope \u8fd0\u884c\u65f6\u4f1a\u5f80\u6570\u636e\u96c6\u76ee\u5f55\u5199\u5165\u7f13\u5b58\u7d22\u5f15\u6587\u4ef6\uff0c\u53ea\u8bfb\u6302\u8f7d\u4f1a\u5bfc\u81f4\u8fd0\u884c\u5931\u8d25\u3002\u6240\u6709 -v<\/code> \u6302\u8f7d\u547d\u4ee4\u4e2d\u8bf7\u53bb\u6389 :ro<\/code>\u3002<\/p>\n
        \u8fd9\u4e2a\u5751\u6bd4\u8f83\u9690\u853d\u2014\u2014\u76f4\u89c9\u4e0a\u6570\u636e\u96c6\u662f\u53ea\u8bfb\u7684\uff0c\u4f46\u5b9e\u9645\u4e0a Arrow\/Parquet \u683c\u5f0f\u7684\u6570\u636e\u96c6\u9996\u6b21\u52a0\u8f7d\u65f6\u6846\u67b6\u4f1a\u751f\u6210\u7d22\u5f15\u7f13\u5b58\uff0c\u5199\u5728\u6570\u636e\u96c6\u76ee\u5f55\u672c\u8eab\u3002\u5982\u679c\u786e\u5b9e\u4e0d\u60f3\u8ba9\u6570\u636e\u96c6\u76ee\u5f55\u88ab\u4fee\u6539\uff0c\u53ef\u4ee5 cp -r<\/code> \u4e00\u4efd\u526f\u672c\u5230\u53ef\u5199\u4f4d\u7f6e\u518d\u6302\u8f7d\u3002<\/p>\n
        8.3 evalscope \u547d\u4ee4\u884c\u89c4\u8303<\/h3>\n
        \u5b98\u65b9\u53ea\u4f7f\u7528 evalscope eval<\/code> + CLI \u53c2\u6570\u3002--config<\/code> \u4e0d\u662f eval<\/code> \u5b50\u547d\u4ee4\u7684\u53c2\u6570\uff0c\u7528\u4e86\u4f1a\u62a5 unrecognized arguments<\/code>\u3002<\/p>\n
        --eval-config<\/code> \u7528\u4e8e\u975e Native \u540e\u7aef\uff08OpenCompass\/VLMEvalKit\uff09\u7684\u914d\u7f6e\u6587\u4ef6\u8def\u5f84\uff0c\u4e0d\u662f\u4efb\u52a1\u914d\u7f6e\u3002\u5982\u679c\u4f60\u5728 YAML \u91cc\u5199\u4e86 datasets<\/code>\u3001model<\/code> \u7b49\u5b57\u6bb5\uff0c\u7136\u540e\u901a\u8fc7 --eval-config<\/code> \u4f20\u5165\uff0c\u8fd9\u4e9b\u5b57\u6bb5\u4f1a\u88ab\u5f53\u6210\u540e\u7aef\u914d\u7f6e\u7684\u4e00\u90e8\u5206\u5b58\u50a8\uff0c\u9876\u5c42\u7684 datasets<\/code> \u4ecd\u7136\u4e3a\u7a7a \u2192 0 \u6761 benchmark\u3002<\/p>\n
        8.4 \u63a8\u8350\u7528\u6cd5\uff1adataset_id \u6307\u5b9a\u672c\u5730\u8def\u5f84<\/h3>\n
        \u5b98\u65b9\u6587\u6863\u63a8\u8350\u7528 --dataset-args<\/code> \u7684 dataset_id<\/code> \u53c2\u6570\u76f4\u63a5\u6307\u5b9a\u672c\u5730\u8def\u5f84\u3002\u65e7\u7248\u7684 local_path<\/code> \u53c2\u6570\u5df2\u5e9f\u5f03\u3002<\/p>\n
        <\/span>--dataset-args <\/span>'{"mmlu": {"dataset_id": "\/path\/to\/data", "few_shot_num": 5}}'<\/span>\n<\/code><\/pre><\/div>\n\n
        \u8fd9\u79cd\u65b9\u5f0f\u5b8c\u5168\u79bb\u7ebf\u3001\u4e0d\u7ecf\u8fc7 ModelScope Hub \u7f13\u5b58\u673a\u5236\uff0c\u907f\u514d\u4e86\u8def\u5f84\u5339\u914d\u5931\u8d25\u548c\u81ea\u52a8\u8054\u7f51\u4e0b\u8f7d\u7684\u95ee\u9898\u3002<\/p>\n
        8.5 BFCL-v4 \u6570\u636e\u6765\u6e90\u786e\u8ba4<\/h3>\n
        \u5173\u4e8e BFCL \u6570\u636e\u5230\u5e95\u9700\u4e0d\u9700\u8981\u989d\u5916\u4e0b\u8f7d\uff0c\u8d70\u4e86\u4e00\u6bb5\u5f2f\u8def\uff1a<\/p>\n
          \n
        • \u6700\u521d\u8ba4\u4e3a\"\u5185\u7f6e\u5728 bfcl-eval \u5305\u4e2d\"\u4e0d\u9700\u8981\u989d\u5916\u5904\u7406<\/li>\n
        • \u540e\u5c1d\u8bd5\u4ece HuggingFace \u9884\u4e0b\u8f7d\uff0c\u53d1\u73b0 BFCL \u6570\u636e\u96c6\u662f gated repo\uff08\u9700\u8981 HF token\uff09\uff0c\u6240\u6709 candidate repo ID \u90fd\u8fd4\u56de 404\/401<\/li>\n
        • \u6700\u7ec8\u901a\u8fc7 Docker \u5bb9\u5668\u5b9e\u9645\u8fd0\u884c evalscope eval \u9a8c\u8bc1\uff0cbfcl-eval<\/code> \u5305\u5185 data\/<\/code> \u76ee\u5f55\u542b 12MB JSON \u6d4b\u8bd5\u6570\u636e\uff0c\u5168\u90e8 18 \u4e2a subset \u8986\u76d6<\/li>\n<\/ul>\n
          \u6240\u4ee5 BFCL-v4 \u8bc4\u6d4b\u5b8c\u5168\u79bb\u7ebf\u53ef\u7528\u3002\u5982\u679c\u672a\u6765 evalscope \u5347\u7ea7\u6539\u53d8\u4e86\u6570\u636e\u52a0\u8f7d\u65b9\u5f0f\uff0c\u53ef\u80fd\u9700\u8981\u91cd\u65b0\u8bc4\u4f30\u3002<\/p>\n
          8.6 \u6570\u636e\u96c6\u5fc5\u987b\u5168\u90e8\u4f20\u8f93<\/h3>\n
          NPU \u8282\u70b9\u4e0a\u5982\u679c\u53ea\u4f20\u4e86\u90e8\u5206\u6570\u636e\u96c6\uff08\u6bd4\u5982\u53ea\u6709 humaneval\uff09\uff0cevalscope \u68c0\u6d4b\u5230\u5176\u4ed6\u6570\u636e\u96c6\u7f3a\u5931\u4f1a\u5c1d\u8bd5\u4ece modelscope.cn \u8054\u7f51\u4e0b\u8f7d \u2192 DNS \u89e3\u6790\u5931\u8d25 \u2192 NameResolutionError<\/code>\u3002\u786e\u4fdd\u6240\u6709\u5728 DATASETS<\/code> \u4e2d\u5217\u51fa\u7684\u6570\u636e\u96c6\u90fd\u5df2 scp \u5230 NPU\u3002<\/p>\n
          \u4e5d\u3001\u6ce8\u610f\u4e8b\u9879<\/h2>\n
            \n
          1. \u6570\u636e\u96c6\u4e0d\u5728\u955c\u50cf\u91cc<\/strong>\uff1a\u5fc5\u987b -v<\/code> \u6302\u8f7d\uff0c\u5426\u5219 evalscope \u5c1d\u8bd5\u8054\u7f51\u4e0b\u8f7d\u800c\u5931\u8d25<\/li>\n
          2. \u4e0d\u8981\u7528 :ro<\/code><\/strong>\uff1a\u6302\u8f7d\u4e0d\u52a0 :ro<\/code>\uff0cEvalScope \u8fd0\u884c\u65f6\u4f1a\u5199\u7f13\u5b58<\/li>\n
          3. SWE-bench \u9700\u8981 Docker socket<\/strong>\uff1a-v \/var\/run\/docker.sock:\/var\/run\/docker.sock<\/code><\/li>\n
          4. BFCL \u79bb\u7ebf\u53ef\u7528<\/strong>\uff1a\u6570\u636e\u5185\u7f6e\u5728 bfcl-eval \u5305\u4e2d<\/li>\n
          5. torch\/NVIDIA \u5305<\/strong>\uff1a\u955c\u50cf\u5185\u662f ARM64 CPU \u7248 torch\uff0c\u8bc4\u6d4b\u8d70 HTTP API \u4e0d\u4f9d\u8d56 GPU<\/li>\n
          6. \u5185\u5b58\u5efa\u8bae<\/strong>\uff1a10GB \u955c\u50cf + \u6570\u636e\u96c6\u52a0\u8f7d + \u6a21\u578b\u63a8\u7406\uff0c\u5efa\u8bae 910B >= 32GB \u5185\u5b58<\/li>\n
          7. \u6b63\u5f0f\u8dd1\u53bb\u6389 LIMIT<\/code><\/strong>\uff1a\u4e0d\u8bbe\u6b64\u73af\u5883\u53d8\u91cf\u5373\u8dd1\u5168\u91cf<\/li>\n<\/ol>\n
            \u7b80\u8bb0\u3002<\/p>\n
            \u5341\u3001\u6784\u5efa\u94fe\u8def\u8e29\u5751\u8865\u8bb0\uff08EvalScope 1.8\/1.9 + \u56fd\u5185\u7f51\u7edc\uff09<\/h2>\n
            \u524d\u9762\u51e0\u8282\u57fa\u4e8e EvalScope 1.7.1\u3002\u5347\u7ea7\u5230 1.8\/1.9 \u540e\uff0c\u4f9d\u8d56\u94fe\u6709\u53d8\u5316\uff0c\u52a0\u4e0a\u56fd\u5185\u7f51\u7edc\u73af\u5883\uff0c\u91cd\u5efa\u955c\u50cf\u65f6\u53c8\u8e29\u4e86\u4e00\u4e32\u5751\u3002\u672c\u8282\u628a\u5b8c\u6574\u7684\u6784\u5efa\u94fe\u8def\u4fee\u590d\u8bb0\u5f55\u4e0b\u6765\uff0c\u4f5c\u4e3a\u8865\u5145\u3002<\/p>\n
            10.1 pip \u76f4\u8fde pypi.org \u88ab TLS \u6390\u65ad \u2192 \u6362\u6e05\u534e\u6e90<\/h3>\n
            \u73b0\u8c61<\/strong>\uff1a\u6784\u5efa\u5361\u5728 pip \u88c5\u5305\uff0c\u53cd\u590d\u91cd\u8bd5\u540e\u5931\u8d25\uff1a<\/p>\n
            <\/span>WARNING: Retrying ... SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol'))\nERROR: Could not find a version that satisfies the requirement evalscope[swe-bench]>=1.8.0 (from versions: none)\nERROR: No matching distribution found for evalscope[swe-bench]>=1.8.0\n<\/code><\/pre><\/div>\n\n
            \u539f\u56e0<\/strong>\uff1a\u5bb9\u5668\u5185\u76f4\u8fde pypi.org<\/code> \u7684 TLS \u8fde\u63a5\u88ab\u4e2d\u9014\u91cd\u7f6e\uff0c\u4e94\u6b21\u91cd\u8bd5\u5168\u6302\u3002<\/p>\n
            \u4fee\u6b63<\/strong>\uff1aDockerfile \u91cc\u88c5\u5305\u524d\u5148\u914d\u7f6e\u6e05\u534e\u955c\u50cf\u6e90\u3002pip config set<\/code> \u4f1a\u5199\u5165 \/root\/.config\/pip\/pip.conf<\/code>\uff0c\u5bf9\u540e\u7eed\u6240\u6709 RUN<\/code> \u7684 pip \u90fd\u751f\u6548\uff08\u5305\u62ec\u5355\u72ec\u88c5 bfcl\u3001tau-bench \u90a3\u51e0\u6b65\uff09\u3002<\/p>\n
            <\/span>RUN<\/span> <\/span>pip <\/span>config <\/span>set<\/span> <\/span>global.index-url <\/span>https:\/\/pypi.tuna.tsinghua.edu.cn\/simple <\/span>&&<\/span> <\/span>\\<\/span>\n    <\/span>pip <\/span>config <\/span>set<\/span> <\/span>global.trusted-host <\/span>pypi.tuna.tsinghua.edu.cn <\/span>&&<\/span> <\/span>\\<\/span>\n    <\/span>pip <\/span>install <\/span>--upgrade <\/span>pip <\/span>&&<\/span> <\/span>pip <\/span>install <\/span>-r <\/span>\/workspace\/requirements.txt\n<\/code><\/pre><\/div>\n\n
            10.2 datasets \u7248\u672c\u51b2\u7a81\uff08ResolutionImpossible\uff0c\u522b\u88ab fsspec \u5e26\u504f\uff09<\/h3>\n
            \u73b0\u8c61<\/strong>\uff1a\u6362\u4e0a\u955c\u50cf\u6e90\u540e\uff0cpip \u80fd\u8fde\u4e86\uff0c\u4f46\u62a5 ResolutionImpossible<\/code>\uff0c\u9519\u8bef\u4fe1\u606f\u6307\u5411 fsspec\uff1a<\/p>\n
            <\/span>The<\/span> <\/span>conflict<\/span> <\/span>is<\/span> <\/span>caused<\/span> <\/span>by<\/span>:<\/span>\n    <\/span>datasets<\/span> <\/span>2.18.0<\/span> <\/span>depends<\/span> <\/span>on<\/span> <\/span>fsspec<\/span><=<\/span>2024.2.0<\/span> <\/span>and<\/span> <\/span>>=<\/span>2023.1.0<\/span>\n    <\/span>fsspec<\/span>[<\/span>http<\/span>]<\/span> <\/span>2024.2.0<\/span> <\/span>depends<\/span> <\/span>on<\/span> <\/span>fsspec<\/span> <\/span>2024.2.0<\/span>\nERROR<\/span>:<\/span> <\/span>ResolutionImpossible<\/span>\n<\/code><\/pre><\/div>\n\n
            \u771f\u56e0<\/strong>\uff1a\u8ddf fsspec \u6ca1\u5173\u7cfb\u3002evalscope[swe-bench]<\/code> \u4f9d\u8d56 modelscope[datasets]>=1.34<\/code>\uff0c\u800c modelscope \u7684 datasets<\/code> extra \u786c\u6027\u8981\u6c42 datasets>=4.0.0<\/code>\uff1b\u4f46 requirements \u91cc pin \u7684\u662f datasets<3.0.0<\/code>\u2014\u2014\u4e24\u8005\u6839\u672c\u65e0\u89e3\u3002pip \u4e3a\u4e86\u6ee1\u8db3 <3.0.0<\/code>\uff0c\u628a modelscope \u4ece 1.38 \u4e00\u8def\u56de\u6eaf\u5230 1.34\uff0c\u6700\u540e\u5728 fsspec \u7248\u672c\u4e0a\u649e\u5899\u62a5\u9519\uff0c\u9519\u8bef\u4fe1\u606f\u6781\u5177\u8bef\u5bfc\u6027\u3002<\/p>\n
            \u6392\u67e5\u6280\u5de7<\/strong>\uff1a\u4e0e\u5176\u7ffb pip \u90a3\u4e00\u5927\u5806\u56de\u6eaf\u65e5\u5fd7\uff0c\u4e0d\u5982\u76f4\u63a5\u67e5\u4f9d\u8d56\u5143\u6570\u636e\uff0c\u4e00\u773c\u5b9a\u4f4d\uff1a<\/p>\n
            <\/span>curl <\/span>-s <\/span>https:\/\/pypi.org\/pypi\/modelscope\/json <\/span>|<\/span> <\/span>python3 <\/span>-c <\/span>\\<\/span>\n <\/span>"import sys,json;[print(r) for r in json.load(sys.stdin)['info']['requires_dist'] if 'datasets' in r.lower()]"<\/span>\n# \u2192 datasets<=4.8.4,>=4.0.0; extra == "datasets"<\/span>\n<\/code><\/pre><\/div>\n\n
            \u4fee\u6b63<\/strong>\uff1arequirements.txt \u628a datasets \u653e\u5f00\u5230 modelscope \u8981\u6c42\u7684\u533a\u95f4\uff08swebench 4.1.0 \u53ea\u5199 datasets<\/code> \u65e0\u4e0a\u9650\uff0c\u517c\u5bb9 4.x\uff09\uff1a<\/p>\n
            <\/span>datasets>=4.0.0,<=4.8.4\n<\/code><\/pre><\/div>\n\n
            10.3 tau-bench \u7684 GitHub \u5b89\u88c5\u5728\u56fd\u5185\u4e0d\u53ef\u9760 \u2192 vendored \u8fdb\u4e0a\u4e0b\u6587<\/h3>\n
            \u73b0\u8c61<\/strong>\uff1a\u955c\u50cf\u5185 pip install<\/code> \u524d\u7684 git clone<\/code>\uff0c\u7b49 127 \u79d2\u540e\u5931\u8d25\uff1a<\/p>\n
            <\/span>Cloning into '\/workspace\/tau-bench'...\nfatal: unable to access 'https:\/\/github.com\/sierra-research\/tau-bench\/':\n  GnuTLS recv error (-110): The TLS connection was non-properly terminated.\n<\/code><\/pre><\/div>\n\n
            GitHub \u76f4\u8fde\u88ab\u6390\u3002<\/p>\n
            \u4fee\u6b63<\/strong>\uff1a\u8fd9\u672c\u6765\u5c31\u662f\"\u79bb\u7ebf\u955c\u50cf\"\uff0c\u7d22\u6027\u628a tau-bench \u76f4\u63a5 vendored \u8fdb\u6784\u5efa\u4e0a\u4e0b\u6587\u2014\u2014\u5bbf\u4e3b\u673a\u7528 GitHub \u52a0\u901f\u955c\u50cf\u514b\u9686\u597d\uff0c\u53bb\u6389 .git<\/code>\uff0cDockerfile \u6539\u6210 COPY<\/code>\uff1a<\/p>\n
            <\/span># \u5bbf\u4e3b\u673a\uff08ghfast.top \u662f\u53ef\u7528\u7684 GitHub \u52a0\u901f\u955c\u50cf\uff09<\/span>\ngit <\/span>clone <\/span>--depth <\/span>1<\/span> <\/span>https:\/\/ghfast.top\/https:\/\/github.com\/sierra-research\/tau-bench <\/span>tau-bench\nrm <\/span>-rf <\/span>tau-bench\/.git\n<\/code><\/pre><\/div>\n\n
            <\/span># Dockerfile\uff1a\u7531 git clone \u6539\u4e3a COPY\uff08tau-bench \u7684 pip \u4f9d\u8d56\u4ecd\u8d70\u6e05\u534e\u6e90\uff0c\u6b63\u5e38\uff09<\/span>\nCOPY<\/span> <\/span>tau-bench <\/span>\/workspace\/tau-bench\nRUN<\/span> <\/span>pip <\/span>install <\/span>-e <\/span>\/workspace\/tau-bench\n<\/code><\/pre><\/div>\n\n
            \u6ce8\u610f<\/strong>\uff1atau-bench \u82e5\u5728\u5de5\u4f5c\u533a\u662f git \u672a\u8ddf\u8e2a\u76ee\u5f55\uff0c\u5bb9\u6613\u88ab\u4e00\u6b21 git clean<\/code> \u6e05\u6389\uff0c\u6784\u5efa\u65f6\u62a5 COPY ... \"\/tau-bench\": not found<\/code>\uff0c\u91cd\u65b0 clone \u4e00\u4efd\u5373\u53ef\u3002<\/p>\n
            10.4 arm64 \u4ea4\u53c9\u6784\u5efa\u5b9e\u6d4b\uff08Intel Mac + QEMU\uff09<\/h3>\n
            <\/span>docker <\/span>buildx <\/span>build <\/span>--platform <\/span>linux\/arm64 <\/span>\\<\/span>\n  <\/span>-f <\/span>Dockerfile.arm64 <\/span>-t <\/span>evalscope-offline:arm64 <\/span>--load <\/span>.\n<\/code><\/pre><\/div>\n\n
              \n
            • \u524d\u63d0<\/strong>\uff1aQEMU binfmt \u5df2\u6ce8\u518c\uff0cdocker run --rm --platform linux\/arm64 alpine uname -m<\/code> \u8fd4\u56de aarch64<\/code> \u5373\u53ef\u3002<\/li>\n
            • \u8017\u65f6<\/strong>\uff1aIntel Mac \u4e0a\u5168\u7a0b QEMU \u6a21\u62df\uff0c\u7ea6 50 \u5206\u949f\uff08\u4e0b\u8f7d + \u7f16\u8bd1 sdist + \u88c5 torch\/CUDA \u90fd\u5728\u6a21\u62df\u5c42\uff0c\u6162\uff09\u3002<\/li>\n
            • \u5173\u952e\u7ed3\u8bba\uff08\u7ea0\u6b63\u4e00\u4e2a\u5e38\u89c1\u8bef\u89e3\uff09<\/strong>\uff1aarm64 \u4e0a\u7684 torch \u5e76\u4e0d\u662f<\/strong> CPU \u7248\uff0c\u5b83\u7167\u6837\u62c9\u6574\u5957 NVIDIA CUDA \u4f9d\u8d56\u2014\u2014\u4f46\u8fd9\u4e9b\u5305\u90fd\u6709 aarch64 \u7684 wheel\uff08nvidia_cublas-\u2026-manylinux_2_27_aarch64.whl<\/code> \u7b49\uff09\uff0c\u6240\u4ee5\u4e0d\u4f1a\u51fa\u73b0\"x86-only CUDA wheel \u5bfc\u81f4 arm64 \u6784\u5efa\u5931\u8d25\"\u7684\u60c5\u51b5\u3002\u6700\u7ec8\u88c5\u7684\u662f torch 2.12.1+cu130<\/code>\u3002<\/li>\n<\/ul>\n
              \u9a8c\u8bc1<\/strong>\uff1a<\/p>\n
              <\/span>docker <\/span>image <\/span>inspect <\/span>evalscope-offline:arm64 <\/span>--format <\/span>'{{.Architecture}}'<\/span>   <\/span># arm64<\/span>\ndocker <\/span>run <\/span>--rm <\/span>--platform <\/span>linux\/arm64 <\/span>evalscope-offline:arm64 <\/span>python <\/span>-c <\/span>\\<\/span>\n <\/span>"import platform,evalscope,datasets,torch; \\<\/span>\n  print(platform.machine(), evalscope.__version__, datasets.__version__, torch.__version__)"<\/span>\n# aarch64 1.9.0 4.8.4 2.12.1+cu130<\/span>\n<\/code><\/pre><\/div>\n\n
              \u955c\u50cf 6.8GB\uff0c\u5bb9\u5668\u5185 machine=aarch64<\/code>\uff0cevalscope \/ datasets \/ swebench \/ bfcl_eval \/ tau_bench \/ torch \u5168\u90e8 import \u6b63\u5e38\u3002<\/p>\n
              10.5 Docker Desktop \u73af\u5883\u5c42\u9762\u7684\u4e24\u4e2a\u5751<\/h3>\n
                \n
              • \u5f15\u64ce\u5047\u6b7b<\/strong>\uff1adocker<\/code> \u5ba2\u6237\u7aef\u6b63\u5e38\uff0c\u4f46 daemon \u8fd4\u56de 500 ... \/_ping<\/code>\uff0c\u65e5\u5fd7\u91cc apiproxy<\/code> \u53cd\u590d dialing 192.168.65.7:2375 ... no route to host<\/code>\u2014\u2014VM \u7f51\u7edc\u6302\u4e86\u3002\u5f7b\u5e95\u9000\u51fa\u5e76\u6740\u5e72\u51c0\u8fdb\u7a0b\u518d\u91cd\u542f\u53ef\u6062\u590d\uff1a<\/li>\n<\/ul>\n
                <\/span>osascript <\/span>-e <\/span>'quit app "Docker"'<\/span>\npkill <\/span>-f <\/span>com.docker.backend;<\/span> <\/span>pkill <\/span>-f <\/span>"Docker Desktop"<\/span>;<\/span> <\/span>pkill <\/span>-f <\/span>com.docker.virtualization\nopen <\/span>-a <\/span>Docker\n<\/code><\/pre><\/div>\n\n
                  \n
                • \u62c9\u57fa\u7840\u955c\u50cf\u5931\u8d25<\/strong>\uff1aconnecting to 127.0.0.1:18888: connection refused<\/code>\u2014\u2014Docker \u914d\u4e86\u672c\u5730\u4ee3\u7406\uff08Clash \u4e4b\u7c7b\uff09\uff0c\u4ee3\u7406\u6ca1\u5f00\u65f6\u4efb\u4f55 pull<\/code> \u90fd\u5931\u8d25\u3002\u8981\u4e48\u5f00\u4ee3\u7406\uff0c\u8981\u4e48\u5728 Docker Desktop \u2192 Settings \u2192 Resources \u2192 Proxies \u91cc\u5173\u6389\u3002\u53e6\u5916 ~\/.docker\/daemon.json<\/code> \u7684 registry-mirrors<\/code> \u522b\u585e\u5360\u4f4d\u7b26\uff08\u6bd4\u5982 https:\/\/your_preferred_mirror<\/code>\uff09\u548c\u4e00\u5806\u5931\u6548\/\u91cd\u590d\u5730\u5740\uff0c\u4f1a\u62d6\u6162\u5e76\u5076\u53d1\u5931\u8d25\u3002<\/li>\n
                • \u5c0f\u63d0\u793a<\/strong>\uff1amacOS \u81ea\u5e26\u6ca1\u6709 timeout<\/code> \u547d\u4ee4\uff08\u90a3\u662f coreutils \u7684 gtimeout<\/code>\uff09\uff0c\u5199\u7b49\u5f85\/\u8d85\u65f6\u811a\u672c\u65f6\u5bb9\u6613\u8e29\u7a7a\u3002<\/li>\n<\/ul>\n
                  \u8865\u8bb0\u5b8c\u6bd5\u3002<\/p>\n
                  10.6 \u5192\u70df\u6d4b\u8bd5\u4e2d\u53d1\u73b0\u7684\u989d\u5916\u5751<\/h3>\n
                  \u955c\u50cf\u6784\u5efa\u5b8c\u6210\u540e\uff0c\u5728 Intel Mac \u4e0a\u7528 QEMU \u6a21\u62df\u8dd1\u4e86\u4e00\u6b21\u5b8c\u6574\u7684\u5192\u70df\u9a8c\u8bc1\uff08mock API + \u5168\u90e8 benchmark\uff09\uff0c\u53c8\u8e29\u51fa\u4e09\u4e2a\u5751\uff0c\u5747\u5df2\u5728\u9879\u76ee\u548c Dockerfile \u4e2d\u4fee\u590d\u3002<\/p>\n
                  \u5751\u4e00\uff1averify_offline.sh \u81ea\u5e26 :ro<\/code> \u53ea\u8bfb\u6302\u8f7d<\/strong><\/p>\n
                  \u9879\u76ee\u81ea\u5e26\u7684 verify_offline.sh<\/code> \u628a modelscope_cache<\/code> \u6302\u6210\u4e86 :ro<\/code>\uff08\u53ea\u8bfb\uff09\u3002\u4f46 evalscope 1.9.0 \u52a0\u8f7d mmlu \u65f6\u4f1a\u5c1d\u8bd5\u5220\u9664 dataset_infos.json<\/code> \u6587\u4ef6\u2014\u2014\u53ea\u8bfb\u6302\u8f7d\u76f4\u63a5 OSError: Read-only file system<\/code> \u2014\u2014\u5b8c\u5168\u590d\u73b0\u4e86\u7b2c\u516b\u8282\u535a\u6587\u91cc\"\u4e0d\u8981\u7528 :ro\"\u7684\u90a3\u6761\u3002scripts\/verify_offline.sh<\/code> \u5df2\u53bb\u6389 :ro<\/code>\u3002<\/p>\n
                  \u5751\u4e8c\uff1a\u81ea\u5b9a\u4e49 DNS \u5bfc\u81f4 host.docker.internal<\/code> \u89e3\u6790\u5931\u8d25<\/strong><\/p>\n
                  ~\/ .docker\/daemon.json<\/code> \u81ea\u5b9a\u4e49\u4e86 dns: [114.114.114.114, 8.8.8.8]<\/code>\uff0c\u8986\u76d6\u4e86 Docker Desktop \u5185\u7f6e\u7684 DNS \u89e3\u6790\uff0c\u5bfc\u81f4\u5bb9\u5668\u91cc host.docker.internal<\/code> \u2192 Name or service not known<\/code> \u2192 mock API \u8fde\u4e0d\u4e0a \u2192 APIConnectionError<\/code>\u3002\u4fee\u590d\uff1a\u7ed9 eval \u5bb9\u5668\u7684 docker run<\/code> \u52a0 --add-host=host.docker.internal:host-gateway<\/code>\uff08\u76f4\u63a5\u5199 \/etc\/hosts<\/code>\uff0c\u7ed5\u8fc7 DNS\uff09\u3002verify_offline.sh<\/code> \u5df2\u6dfb\u52a0\u6b64\u53c2\u6570\u3002<\/p>\n
                  \u5751\u4e09\uff1abfcl-eval \u7f3a\u5c11 soundfile \u4f9d\u8d56<\/strong><\/p>\n
                  BFCL-v4 \u7684\u63a8\u7406\u9636\u6bb5\u80fd\u8dd1\u901a\uff0c\u4f46\u8fdb\u5165\u8bc4\u5206\u9636\u6bb5\u65f6\u62a5 ModuleNotFoundError: No module named 'soundfile'<\/code>\u3002\u94fe\u8def\u662f bfcl_eval<\/code> \u8bc4\u5206 \u2192 qwen_agent<\/code> \u2192 soundfile<\/code>\uff0c\u8fd9\u4e2a\u4f9d\u8d56\u5728 bfcl-eval==2025.10.27.1<\/code> \u7684\u5143\u6570\u636e\u91cc\u662f\u95f4\u63a5\u7684\uff08\u901a\u8fc7 qwen_agent<\/code>\uff09\uff0c\u6240\u4ee5 evalscope[bfcl]<\/code> \u5b89\u88c5\u65f6\u4e0d\u4f1a\u81ea\u52a8\u62c9\u8fdb\u6765\u3002<\/p>\n
                  \u597d\u6d88\u606f\u662f PySoundFile 0.14+ \u7684 aarch64 wheel \u81ea\u5e26 libsndfile\uff0c\u4e0d\u9700\u8981 apt \u88c5\u7cfb\u7edf\u5e93\u2014\u2014\u53ea\u9700 pip install soundfile<\/code> \u5373\u53ef\u3002Dockerfile \u5c3e\u90e8\u8ffd\u52a0\u4e86\u4e00\u4e2a RUN pip install soundfile<\/code> \u5c42\uff08\u653e\u6700\u540e\u4e0d\u5f71\u54cd\u91cd\u578b\u7f13\u5b58\uff09\u3002<\/p>\n
                  \u6b64\u5916\uff0cscripts\/run_agent_eval.sh<\/code> \u7684 build_dataset_args<\/code> \u51fd\u6570\u6709\u4e2a JSON \u6784\u9020 bug\uff1a\u628a\u5df2\u5b8c\u6574\u7684 JSON \u5bf9\u8c61\u53c8\u5957\u4e86\u4e00\u5c42\u5927\u62ec\u53f7\uff0c\u4ea7\u751f {{...}}<\/code> \u975e\u6cd5 JSON \u2192 JSONDecodeError<\/code>\u3002\u5df2\u7528 Python \u5408\u5e76\u66ff\u4ee3\u5b57\u7b26\u4e32\u62fc\u63a5\uff0c\u811a\u672c\u4e2d\u4e5f\u5df2\u4fee\u590d\u3002<\/p>\n
                  \u5168\u6587\u8865\u8bb0\u5b8c\u6bd5\u3002<\/p>\n
                  10.7 datasets 4.0.0 Value dtype \u517c\u5bb9\u8865\u4e01\uff082026-07-13\uff09<\/h3>\n
                  \u73b0\u8c61<\/strong>\uff1aSWE-bench \u8bc4\u6d4b\u5728\u52a0\u8f7d oracle \u6570\u636e\u96c6\u65f6\u62a5\u9519\uff1a<\/p>\n
                  <\/span>TypeError<\/span>:<\/span> <\/span>Value<\/span>.<\/span>__init__<\/span>()<\/span> <\/span>missing<\/span> <\/span>1<\/span> <\/span>required<\/span> <\/span>positional<\/span> <\/span>argument<\/span>:<\/span> <\/span>'<\/span>dtype<\/span>'<\/span>\n<\/code><\/pre><\/div>\n\n
                  \u5b8c\u6574\u8c03\u7528\u6808\uff1a<\/p>\n
                  <\/span>File<\/span> <\/span>"datasets\/info.py"<\/span>,<\/span> <\/span>line<\/span> <\/span>170<\/span>,<\/span> <\/span>in<\/span> <\/span>__post_init__<\/span>\n  <\/span>self<\/span>.<\/span>features<\/span> <\/span>=<\/span> <\/span>Features<\/span>.<\/span>from_dict<\/span>(<\/span>self<\/span>.<\/span>features<\/span>)<\/span>\nFile<\/span> <\/span>"datasets\/features\/features.py"<\/span>,<\/span> <\/span>line<\/span> <\/span>1583<\/span>,<\/span> <\/span>in<\/span> <\/span>generate_from_dict<\/span>\n  <\/span>return<\/span> <\/span>class_type<\/span>(<\/span>**<\/span>{<\/span>k<\/span>:<\/span> <\/span>v<\/span> <\/span>for<\/span> <\/span>k<\/span>,<\/span> <\/span>v<\/span> <\/span>in<\/span> <\/span>obj<\/span>.<\/span>items<\/span>()<\/span> <\/span>if<\/span> <\/span>k<\/span> <\/span>in<\/span> <\/span>field_names<\/span>})<\/span>\nTypeError<\/span>:<\/span> <\/span>Value<\/span>.<\/span>__init__<\/span>()<\/span> <\/span>missing<\/span> <\/span>1<\/span> <\/span>required<\/span> <\/span>positional<\/span> <\/span>argument<\/span>:<\/span> <\/span>'<\/span>dtype<\/span>'<\/span>\n<\/code><\/pre><\/div>\n\n
                  \u539f\u56e0<\/strong>\uff1a\u8fd9\u5176\u5b9e\u662f 10.2 \u8282\u4fee\u590d ResolutionImpossible \u7684\u526f\u4f5c\u7528\u3002\u4e3a\u4e86\u6ee1\u8db3 modelscope[datasets]>=1.34<\/code> \u7684\u8981\u6c42\uff0c\u6211\u4eec\u628a datasets \u4ece <3.0.0<\/code> \u5347\u5230\u4e86 >=4.0.0<\/code>\u3002\u4f46 datasets 4.0.0 \u91cd\u505a\u4e86 Value<\/code> feature \u7c7b\u7684 __init__<\/code>\uff0c\u628a dtype<\/code> \u6539\u6210\u4e86\u5fc5\u4f20\u4f4d\u7f6e\u53c2\u6570\u3002\u800c ModelScope \u4e0a\u7f13\u5b58\u7684\u65e7\u6570\u636e\u96c6\uff08\u5982 princeton-nlp\/SWE-bench_oracle<\/code>\uff09\u662f\u7528\u65e7\u7248 datasets \u5e8f\u5217\u5316\u7684\uff0c\u5143\u6570\u636e\u91cc\u6ca1\u6709 dtype<\/code> \u5b57\u6bb5\u3002\u65b0 datasets \u52a0\u8f7d\u65e7\u5143\u6570\u636e\u65f6\u76f4\u63a5\u70b8\u3002<\/p>\n
                  \u8fd9\u4e2a\u9519\u8bef\u7b2c\u4e00\u6b21\u51fa\u73b0\u5728 7 \u6708 7 \u65e5\u2014\u2014\u5f53\u65f6\u5fd9\u7740\u4fee ResolutionImpossible \u548c tau-bench vendored\uff0c\u6ca1\u6ce8\u610f\u5230 datasets \u7248\u672c\u5347\u7ea7\u7684\u8fde\u5e26\u5f71\u54cd\u30027 \u6708 13 \u65e5\u91cd\u65b0\u8dd1 SWE-bench \u65f6\u624d\u6b63\u5f0f\u786e\u8ba4\u5e76\u4fee\u590d\u3002<\/p>\n
                  \u4fee\u6b63<\/strong>\uff1a\u5199\u4e86\u4e00\u4e2a monkey-patch \u811a\u672c patch_datasets_compat.py<\/code>\uff0c\u62e6\u622a datasets.features.features.generate_from_dict<\/code>\u3002\u5f53\u9047\u5230\u65e7\u683c\u5f0f\u7684 Value<\/code> \u5bf9\u8c61\uff08_type=Value<\/code> \u4f46\u7f3a\u5c11 dtype<\/code>\uff09\u65f6\uff0c\u81ea\u52a8\u8865 dtype: \"string\"<\/code> \u4f5c\u4e3a\u9ed8\u8ba4\u503c\uff1a<\/p>\n
                  <\/span>import<\/span> <\/span>datasets.features.features<\/span> <\/span>as<\/span> <\/span>f<\/span>\n\n_orig<\/span> =<\/span> f<\/span>.<\/span>generate_from_dict<\/span>\n\ndef<\/span> <\/span>_patched<\/span>(<\/span>obj<\/span>):<\/span>\n    if<\/span> isinstance<\/span>(<\/span>obj<\/span>,<\/span> dict<\/span>)<\/span> and<\/span> obj<\/span>.<\/span>get<\/span>(<\/span>"_type"<\/span>)<\/span> ==<\/span> "Value"<\/span> and<\/span> "dtype"<\/span> not<\/span> in<\/span> obj<\/span>:<\/span>\n        obj<\/span> =<\/span> {<\/span>**<\/span>obj<\/span>,<\/span> "dtype"<\/span>:<\/span> "string"<\/span>}<\/span>\n    return<\/span> _orig<\/span>(<\/span>obj<\/span>)<\/span>\n\nf<\/span>.<\/span>generate_from_dict<\/span> =<\/span> _patched<\/span>\n<\/code><\/pre><\/div>\n\n
                  \u4e3a\u4e86\u8ba9\u8865\u4e01\u5728\u4efb\u4f55 Python \u8fdb\u7a0b\u542f\u52a8\u65f6\u81ea\u52a8\u751f\u6548\uff0c\u5728 Dockerfile \u4e2d\u901a\u8fc7 install_patch.py<\/code> \u5c06\u8865\u4e01\u811a\u672c\u5b89\u88c5\u5230 site-packages\/<\/code>\uff0c\u5e76\u521b\u5efa .pth<\/code> \u6587\u4ef6\u89e6\u53d1\u81ea\u52a8 import\uff1a<\/p>\n
                  <\/span># install_patch.py<\/span>\nimport<\/span> <\/span>shutil<\/span>,<\/span> <\/span>site<\/span>,<\/span> <\/span>os<\/span>\nsrc<\/span> =<\/span> '\/workspace\/scripts\/patch_datasets_compat.py'<\/span>\ndst<\/span> =<\/span> os<\/span>.<\/span>path<\/span>.<\/span>join<\/span>(<\/span>site<\/span>.<\/span>getsitepackages<\/span>()[<\/span>0<\/span>],<\/span> 'patch_datasets_compat.py'<\/span>)<\/span>\nshutil<\/span>.<\/span>copy<\/span>(<\/span>src<\/span>,<\/span> dst<\/span>)<\/span>\nwith<\/span> open<\/span>(<\/span>os<\/span>.<\/span>path<\/span>.<\/span>join<\/span>(<\/span>site<\/span>.<\/span>getsitepackages<\/span>()[<\/span>0<\/span>],<\/span> 'patch_datasets_compat.pth'<\/span>),<\/span> 'w'<\/span>)<\/span> as<\/span> f<\/span>:<\/span>\n    f<\/span>.<\/span>write<\/span>(<\/span>'import patch_datasets_compat<\/span>\\n<\/span>'<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
                  .pth<\/code> \u6587\u4ef6\u662f Python \u7684 site \u6a21\u5757\u673a\u5236\u2014\u2014\u653e\u5728 site-packages \u4e0b\u7684 .pth<\/code> \u6587\u4ef6\u4f1a\u88ab Python \u542f\u52a8\u65f6\u81ea\u52a8\u5904\u7406\uff0c\u4ee5 import<\/code> \u5f00\u5934\u7684\u884c\u4f1a\u88ab\u6267\u884c\uff0c\u7b49\u4ef7\u4e8e\u5728 sitecustomize.py<\/code> \u4e2d import\u3002<\/p>\n
                  \u9a8c\u8bc1<\/strong>\uff1a\u6784\u5efa\u540e\u5728\u5bb9\u5668\u5185\u8dd1\uff1a<\/p>\n
                  <\/span>from<\/span> <\/span>datasets.features.features<\/span> <\/span>import<\/span> generate_from_dict<\/span>\nr<\/span> =<\/span> generate_from_dict<\/span>({<\/span>'_type'<\/span>:<\/span> 'Value'<\/span>,<\/span> 'id'<\/span>:<\/span> None<\/span>})<\/span>\nprint<\/span>(<\/span>r<\/span>)<\/span>  # \u2192 Value('string')<\/span>\n<\/code><\/pre><\/div>\n\n
                  \u65e7\u683c\u5f0f\u53cd\u5e8f\u5217\u5316\u6b63\u5e38\uff0c\u4e0d\u518d\u62a5 TypeError\u3002<\/p>\n
                  \u4e3a\u4ec0\u4e48\u4e0d\u7528\u66f4\u7b80\u5355\u7684\u65b9\u6cd5\uff1f<\/strong> \u8003\u8651\u8fc7\u4e09\u4e2a\u65b9\u6848\uff1a<\/p>\n
                    \n
                  • \u6e05\u7406 modelscope \u7f13\u5b58\u91cd\u65b0\u4e0b\u8f7d<\/strong>\uff1a\u4e0d\u884c\u3002910B \u662f\u79bb\u7ebf\u73af\u5883\uff0c\u91cd\u65b0\u4e0b\u8f7d\u8981\u8d70 Mac \u2192 scp \u6d41\u7a0b\uff0c\u800c\u4e14\u6ca1\u6cd5\u4fdd\u8bc1\u65b0\u4e0b\u8f7d\u7684\u5143\u6570\u636e\u683c\u5f0f\u517c\u5bb9\u2014\u2014datasets 4.x \u5199\u5165\u7684\u662f\u65b0\u683c\u5f0f\uff0c\u4f46\u6e90\u6570\u636e\u96c6\u662f\u65e7\u683c\u5f0f\u5b58\u50a8\u7684\uff0c\u95ee\u9898\u4f9d\u7136\u5b58\u5728\u3002<\/li>\n
                  • \u964d\u7ea7 datasets<\/strong>\uff1a\u4e0d\u884c\u3002modelscope>=1.34 \u786c\u6027\u8981\u6c42 datasets>=4.0.0\u3002<\/li>\n
                  • \u76f4\u63a5\u6539 datasets \u6e90\u7801<\/strong>\uff1a\u53ef\u884c\u4f46\u8106\u5f31\uff0cpip \u5347\u7ea7\u4f1a\u8986\u76d6\u3002.pth<\/code> \u81ea\u52a8\u6ce8\u5165\u66f4\u4f18\u96c5\uff0c\u4e0d\u4fb5\u5165\u4e09\u65b9\u5305\u3002<\/li>\n<\/ul>\n
                    \u81f3\u6b64\uff0cEvalScope 1.9.0 + \u79bb\u7ebf arm64 \u955c\u50cf\u7684\u6240\u6709\u6784\u5efa\u548c\u90e8\u7f72\u95ee\u9898\u5df2\u5168\u90e8\u4fee\u590d\u3002<\/p>\n
                    10.8 Dockerfile \u884c\u5185 Python \u591a\u884c\u8bed\u6cd5\u9677\u9631<\/h3>\n
                    \u5199\u8865\u4e01\u5b89\u88c5\u7684\u65f6\u5019\u53c8\u8e29\u4e86\u4e2a Dockerfile \u8bed\u6cd5\u5751\uff1a<\/p>\n
                    <\/span># \u274c \u9519\u8bef\uff1aDocker \u628a\u7b2c\u4e8c\u4e2a import \u5f53\u6210\u4e86\u4e00\u6761\u65b0\u6307\u4ee4<\/span>\nRUN<\/span> <\/span>python3 <\/span>-c <\/span>"<\/span>\nimport <\/span>shutil, <\/span>site, <\/span>os\n...\n"<\/span>\n# \u2192 dockerfile parse error: unknown instruction: import<\/span>\n<\/code><\/pre><\/div>\n\n
                    Dockerfile \u7684 RUN<\/code> \u547d\u4ee4\u4e0d\u652f\u6301\u8de8\u884c\u5b57\u7b26\u4e32\uff08\u9664\u975e\u7528 \\<\/code> \u8fde\u63a5\u6216 HEREDOC \u8bed\u6cd5\uff09\u3002\u4fee\u6b63\u65b9\u6cd5\uff1a\u628a Python \u903b\u8f91\u62bd\u5230\u72ec\u7acb\u811a\u672c install_patch.py<\/code> \u91cc\uff0cDockerfile \u53ea\u8dd1\u4e00\u6761 RUN python3 \/workspace\/scripts\/install_patch.py<\/code>\u3002<\/p>\n
                    \u5b8c\u6574\u8865\u8bb0\u5b8c\u6bd5\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"
                    EvalScope \u79bb\u7ebf\u8bc4\u6d4b\u90e8\u7f72\u7b80\u8bb0\uff08\u534e\u4e3a910B ARM + OpenAI \u517c\u5bb9\u63a5\u53e3\uff09 \u80cc\u666f \u5728\u534e\u4e3a 910B (ARM64)  …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1688","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","jetpack-related-posts":[],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.izhuhn.cn\/index.php\/wp-json\/wp\/v2\/posts\/1688","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.izhuhn.cn\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.izhuhn.cn\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.izhuhn.cn\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.izhuhn.cn\/index.php\/wp-json\/wp\/v2\/comments?post=1688"}],"version-history":[{"count":16,"href":"https:\/\/www.izhuhn.cn\/index.php\/wp-json\/wp\/v2\/posts\/1688\/revisions"}],"predecessor-version":[{"id":1788,"href":"https:\/\/www.izhuhn.cn\/index.php\/wp-json\/wp\/v2\/posts\/1688\/revisions\/1788"}],"wp:attachment":[{"href":"https:\/\/www.izhuhn.cn\/index.php\/wp-json\/wp\/v2\/media?parent=1688"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.izhuhn.cn\/index.php\/wp-json\/wp\/v2\/categories?post=1688"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.izhuhn.cn\/index.php\/wp-json\/wp\/v2\/tags?post=1688"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}