add_generation_prompt=False in Tokenizer.apply_chat_template has no effect

[AndreiMuresanu]

System Info

Hi, I am trying to apply a chat template to an input without the generation prompt. However, setting add_generation_prompt=False appears to have no effect.

Who can help?

@ArthurZucker @Rocketknight1

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Minimal reproducible example:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

chat = [
		{"role": "assistant", "content": "Hello, how are you?"},
	]
formatted_chat = tokenizer.apply_chat_template(chats, add_generation_prompt=False, tokenize=False)
print(formatted_chat)

transformers==4.41.0

Expected behavior

Observed Output:

<|begin_of_text|><|start_header_id|>assistant<|end_header_id|>

Hello, how are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Expected Output:

<|begin_of_text|><|start_header_id|>assistant<|end_header_id|>

Hello, how are you?<|eot_id|>

[Rocketknight1]

Hi @AndreiMuresanu, I just tried to reproduce this and add_generation_prompt=False worked correctly for me. Can you retry your code, and make sure you're updated to the latest version of transformers?

[Taeyoung-Jang]

I have also encountered this issue.

model : Llama-3-8B-Instruct
package: transformers==4.41.1, tokenizer==0.19.1

[Rocketknight1]

Can you try installing transformers from main with pip install git+https://github.com/huggingface/transformers.git? I can't reproduce this issue at all, and I'm curious if it's caused by an old cached version of the model, or some recent change, or something.

[iseesaw]

same problem

[iseesaw]

same problem

Solved. Because I used the initial chat template, which was later updated.

See Fix chat template to add generation prompt only if the option is selected (#9)

[Rocketknight1]

@iseesaw thanks for pointing that one out! It seems like this was an issue with the model's chat template, which has since been resolved, and the issue will be fixed if people redownload the tokenizer. Going to close this issue now, since it's not a bug in transformers itself.