ChatGLM2 Support

[GoHomeToMacDonal]

An implementation of ChatGLM 2 based on vLLM. This implementation adapts PagedAttentionWithRoPE and ParallelLinear layers for model inference.

[simon-mo]

Thank you for the contribution, unfortunately this PR seems to have some merge conflict and ChatGLM3 also came out. Feel free to coordinate the contribution here if you have bandwidth!

#1552

[GoHomeToMacDonal]

@simon-mo Hi, we have resolved the code conflict, and it can be directly merged into the main branch.

As chatglm3 does not change the model structure, this implementation can be directly adopted to chatglm3. Below is the testing code:

from vllm import LLM, SamplingParams

prompts = ["""<|system|>
You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.
<|user|>
Hello
<|assistant|>
"""]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(model="/home/skim/.cache/modelscope/hub/ZhipuAI/chatglm3-6b", trust_remote_code=True)

outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

The output will be Hello! How can I assist you today?

[zhuohan123]

LGTM! Merged with main and fixed a small style issue. The code works with both ChatGLM2 and ChatGLM3 on one GPU in my case. Thank you for your contribution!

[Midnight-719]

hi ,can u tell me how to use it , I still have this problem: AttributeError: 'ChatGLMConfig' object has no attribute 'num_hidden_layers', Currently, I have updated to the latest version of VLLM

[GoHomeToMacDonal]

This is problem is caused by old version of transformers. I suggest upgrading both your transformers package and ChatGLM model to the recent versions.

[Midnight-719]

This is problem is caused by old version of transformers. I suggest upgrading both your transformers package and ChatGLM model to the recent versions.

Yes, I have tried，transformers==4.35.0

[GoHomeToMacDonal]

This is problem is caused by old version of transformers. I suggest upgrading both your transformers package and ChatGLM model to the recent versions.

Yes, I have tried，transformers==4.35.0

Please provide more information of installed packages, and I will try to reproduce your problem later.

[Jeffwan]

@GoHomeToMacDonal If you use other prompts, it shows big difference between the native model.. Did you try more examples?

this is one example, seems it stopped after meeting some token

[GoHomeToMacDonal]

@GoHomeToMacDonal If you use other prompts, it shows big difference between the native model.. Did you try more examples?

this is one example, seems it stopped after meeting some token

I guess you used the default max_tokens=16 in the sampling parameters. I suggest to set max_tokens to a larger value, e.g., 1024. For more details, please refer to vllm/sampling_params.py.

In addition, as ChatGLM3 added some special tokens, e.g., <|system|>, use ChatGLMTokenizer.build_chat_input build the input token ids and feed them into vLLM will generate more stable results.

[Jeffwan]

@GoHomeToMacDonal It is the max_tokens setting issue. adding max_tokens works as expected. It's my first time using openai wrapper, thanks for the advice. BTW, I plan to use lm-sys/FastChat#2622 to build the conv template, I did some test and the result looks equivalent to what ChatGLMTokenizer.build_chat_input generates.

build the input token ids and feed them into vLLM will generate more stable results.
Just curious, does vllm provides the token interface?

[wangruohui]

Hello, I am using ChatGLM2 but it seems sometimes the output is not aligned with huggingface version. Could anyone help to take a look at #1670 ?

[zengzikang]

The latest version already supports GLM. Can GLM3 support official tool calls and other functions? Does it support dialogue function?

[GoHomeToMacDonal]

The latest version already supports GLM. Can GLM3 support official tool calls and other functions? Does it support the dialogue function?

You need to implement the corresponding code for function calls and prompt building. The vllm library focuses on model inference, i.e., it can substitute ChatGLMForConditionalGeneration.generate to llm.generate. I suggest building prompts based on the official ChatGLM 3 repository, and replacing model inference functions, e.g., ChatGLMForConditionalGeneration.chat, ChatGLMForConditionalGeneration.stream_chat with vLLM.

[zengzikang]

The latest version already supports GLM. Can GLM3 support official tool calls and other functions? Does it support the dialogue function?

You need to implement the corresponding code for function calls and prompt building. The vllm library focuses on model inference, i.e., it can substitute ChatGLMForConditionalGeneration.generate to llm.generate. I suggest building prompts based on the official ChatGLM 3 repository, and replacing model inference functions, e.g., ChatGLMForConditionalGeneration.chat, ChatGLMForConditionalGeneration.stream_chat with vLLM.

Using vllm to infer the GLM3 model, the speed is only about 13% faster, is it normal?

[junior-zsy]

@GoHomeToMacDonal This is still not supported for the Chatglm2-6b-32k version ,I have a message for the issue #1725