-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Last Token Embedding not matching #2591
Comments
Hello! If I remember correctly, https://huggingface.co/intfloat/e5-mistral-7b-instruct is not compatible with Sentence Transformers out of the box due to its left-sided tokenizer. The Sentence Transformers Pooler isn't expecting such a tokenizer, so it incorrectly picks the "last" token.
|
@akjindal53244 You need to pass add_eos_token=True when creating the sentence-transformers model
Output:
|
Oh, wow! Great job. Do you think we should make a pull request on https://huggingface.co/intfloat/e5-mistral-7b-instruct to add Then the transformers-based usage snippet can be simplified and we can integrate Sentence Transformers out of the box! cc @intfloat as this might interest you
|
Hi @tomaarsen @satyamk7054 , coincidentally, I was also able to get it working after few hours of opening the issue. I was able to trace the issue by comparing Here is my code:
To load the model: We should add these changes to the repo as the model is widely used :) |
I am using
intfloat/e5-mistral-7b-instruct
model to get last hidden state for my input and compute cosine similarity.I am using a toy example provided at: https://huggingface.co/intfloat/e5-mistral-7b-instruct#usage
Method-1
Code ref: https://huggingface.co/intfloat/e5-mistral-7b-instruct#usage
Output:
[[0.8291366100311279, 0.4797157943248749], [0.46951112151145935, 0.8174349665641785]]
Method-2
Here is my code using
SentenceTransformers
library:Output:
[[0.5713191032409668, 0.20948277413845062], [0.3217913508415222, 0.5535271167755127]]
Output
of both should match but looks like I am missing something obvious. Any help is appreciated!The text was updated successfully, but these errors were encountered: