error occurred when deploying the transformed model using the vllm server

[Wfd567]

@MFX12138
hi, author, I can deploy the converted model offline using vllm, but an error occurs when using the vllm server. May I ask if there is a startup command for the online deployment method

[xueliangyang-oeuler]

@fxmeng hello, We discovery that after transformed to MLA, the related performance is worse. And the transformed model cannot be started by vllm online serving. Is there any good case for it?

[Stanleytowne]

@xueliangyang-oeuler @Wfd567 To help us troubleshoot the "error occurs when using the vllm server" issue, could you please provide the specific command (startup command) you used to launch the vllm server and the full error message it produced?

[Wfd567]

@xueliangyang-oeuler @Wfd567为了帮助我们排查“使用 vllm 服务器时发生错误”的问题，请您提供启动 vllm 服务器的具体命令（启动命令）以及它产生的完整错误消息？
Thank you for your reply @Stanleytowne
error logs:
INFO 10-29 00:03:10 [cuda.py:187] Using Triton MLA backend on V1 engine.
WARNING 10-29 00:03:10 [triton_decode_attention.py:44] The following error message 'operation scheduled before its operands' can be ignored.
INFO 10-29 00:03:10 [gpu_model_runner.py:1276] Starting to load model /data/data_on_test/transMLA/TransMLA-main/scripts/outputs2/qwen2_5-7B-Instruct-tran...
WARNING 10-29 00:03:10 [utils.py:76] LlamaMLAForCausalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
INFO 10-29 00:03:10 [transformers.py:118] Using Transformers backend.
INFO 10-29 00:03:11 [cuda.py:221] Using Flash Attention backend on V1 engine.
ERROR 10-29 00:03:11 [core.py:387] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 151, in init
ERROR 10-29 00:03:11 [core.py:387] old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
ERROR 10-29 00:03:11 [core.py:387] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers.py", line 378, in init
ERROR 10-29 00:03:11 [core.py:387] self.model = TransformersModel(vllm_config=vllm_config, prefix=prefix)
ERROR 10-29 00:03:11 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-29 00:03:11 [core.py:387] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers.py", line 165, in init
ERROR 10-29 00:03:11 [core.py:387] self.attention_instances = self.create_attention_instances()
ERROR 10-29 00:03:11 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-29 00:03:11 [core.py:387] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers.py", line 268, in create_attention_instances
ERROR 10-29 00:03:11 [core.py:387] Attention(
ERROR 10-29 00:03:11 [core.py:387] File "/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py", line 130, in init
ERROR 10-29 00:03:11 [core.py:387] self.impl = impl_cls(num_heads, head_size, scale, num_kv_heads,
ERROR 10-29 00:03:11 [core.py:387] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-29 00:03:11 [core.py:387] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flash_attn.py", line 397, in init
ERROR 10-29 00:03:11 [core.py:387] raise ValueError(
ERROR 10-29 00:03:11 [core.py:387] ValueError: Head size 576 is not supported by FlashAttention. Supported head sizes are: [32, 64, 96, 128, 160, 192, 224, 256]. Set VLLM_USE_V1=0 to use another attention backend.

command:
import subprocess
import os
from vllm_registry import deepseek

def serve_model():
process = subprocess.Popen([
"python", "-m", "vllm.entrypoints.openai.api_server",
"--model", "/data/data_on_test/transMLA/TransMLA-main/scripts/outputs2/qwen2_5-7B-Instruct-tran",
"--trust-remote-code",
"--host", "0.0.0.0",
"--port", "8080",
"--gpu-memory-utilization", "0.9",
"--max-model-len", "4096",
"--tensor-parallel-size", "1"
])
process.wait()

if name == "main":
serve_model()

from the error message, it seems that the registration was not successful, but I have ensured that the model registration in the deepseek.py script has been executed,

[Stanleytowne]

Hi, thanks for providing the details.

The model registration wouldn't be successful using your script. The reason this is happening is due to process isolation.

1. When you run from vllm_registry import deepseek, you register your model in the main script's process.
2. However, subprocess.Popen launches the vLLM server in a completely new, separate process.
3. This new process has no knowledge of the model you registered in the main process, so the registration fails.

Please try replacing your script with the following code:

import os
import sys
import runpy  # Import the runpy module

# 1. First, import your model registration script.
#    This registers your custom model in the *current* process.
from transmla.vllm_registry import deepseek 

def serve_model():
    
    # 2. Build the list of arguments for the vLLM server
    args = [
        "vllm.entrypoints.openai.api_server",  # This simulates the script name
        "--model", [path to your model],
        "--trust-remote-code",
        "--host", "0.0.0.0",
        "--port", "8080",
        "--gpu-memory-utilization", "0.9",
        "--max-model-len", "4096",
        "--tensor-parallel-size", "1"
    ]

    # 3. Assign the arguments to sys.argv
    sys.argv = args

    # 4. Use runpy.run_module to execute the vLLM server.
    #    This runs the module's __main__ block in the *same process*,
    #    so it will find the model you registered in step 1.
    runpy.run_module("vllm.entrypoints.openai.api_server", run_name="__main__")

if __name__ == "__main__":
    serve_model()

[Wfd567]

The issue has been resolved. Thank you for your assistance @Stanleytowne

[Wfd567]

@Stanleytowne
However, after testing, the throughput of the converted Qwen2.5-7B model has slowed down. Both the test script you uploaded and my own test results indicate it is slower. May I ask, is there any reasonable explanation for this test result? The conversion script used was the one from the repository.

[xueliangyang-oeuler]

@Wfd567 pls list your testing results. And are there any methods to improve the performance? @fxmeng Can youl give some suggestions for this?

[Wfd567]

@Stanleytowne this is test results
Running /data/QWen/QWen257B/Qwen2.5-7B-Instruct with input length 1024 and output length 1024
Throughput: 3.71 requests/s, 7586.02 total tokens/s, 3794.31 output tokens/s
Total num prompt tokens: 102330
Total num output tokens: 102400
Running /data/data_on_test/transMLA/TransMLA-main/scripts/outputs3-nv/qwen2_5-7B-Instruct-deepseek with input length 1024 and output length 1024
Throughput: 2.48 requests/s, 5079.47 total tokens/s, 2542.98 output tokens/s
Total num prompt tokens: 102139
Total num output tokens: 102400

Running /data/QWen/QWen257B/Qwen2.5-7B-Instruct with input length 2048 and output length 2048
Throughput: 1.54 requests/s, 6287.89 total tokens/s, 3145.04 output tokens/s
Total num prompt tokens: 204657
Total num output tokens: 204800
Running /data/data_on_test/transMLA/TransMLA-main/scripts/outputs3-nv/qwen2_5-7B-Instruct-deepseek with input length 2048 and output length 2048
Throughput: 1.00 requests/s, 4077.39 total tokens/s, 2040.21 output tokens/s
Total num prompt tokens: 204497
Total num output tokens: 204800

Running /data/QWen/QWen257B/Qwen2.5-7B-Instruct with input length 4096 and output length 4096
Throughput: 0.57 requests/s, 4680.83 total tokens/s, 2341.51 output tokens/s
Total num prompt tokens: 409216
Total num output tokens: 409600
Running /data/data_on_test/transMLA/TransMLA-main/scripts/outputs3-nv/qwen2_5-7B-Instruct-deepseek with input length 4096 and output length 4096
Throughput: 0.34 requests/s, 2778.43 total tokens/s, 1389.85 output tokens/s
Total num prompt tokens: 409223
Total num output tokens: 409600

Running /data/QWen/QWen257B/Qwen2.5-7B-Instruct with input length 8192 and output length 8192
Throughput: 0.15 requests/s, 2433.17 total tokens/s, 1216.80 output tokens/s
Total num prompt tokens: 818914
Total num output tokens: 819200
Running /data/data_on_test/transMLA/TransMLA-main/scripts/outputs3-nv/qwen2_5-7B-Instruct-deepseek with input length 8192 and output length 8192
Throughput: 0.10 requests/s, 1693.67 total tokens/s, 846.93 output tokens/s
Total num prompt tokens: 819018
Total num output tokens: 819200

[radna0]

Were you able to address this issue? @Wfd567