HuggingFaceCrossEncoder Issue: RuntimeError: The expanded size of the tensor (614) must match the existing size (512) at non-singleton dimension 1

[weissenbacherpwc]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from langchain_community.cross_encoders import HuggingFaceCrossEncoder

reranking_model = HuggingFaceCrossEncoder(model_name="cross-encoder/msmarco-MiniLM-L12-en-de-v1")

def load_retriever(embeddings,
collection_name,
CONNECTION_STRING,
use_parent_retriever=cfg.USE_PARENT_RETRIEVER,
use_colbert=cfg.USE_COLBERT,
use_cross_encoder = cfg.USE_CROSS_ENCODER,
reranking_model=None):
# Basic Retriever
if use_parent_retriever == False:
db = load_PG_vectorstore(embeddings=embeddings, COLLECTION_NAME=collection_name, CONNECTION_STRING=CONNECTION_STRING)
retriever = db.as_retriever(search_kwargs={'k': cfg.VECTOR_COUNT, 'score_threshold': cfg.SCORE_THRESHOLD}, search_type="similarity_score_threshold")

# ParentDocument Retriever
elif use_parent_retriever == True:
    print("Using ParentDocumentRetriever")
    retriever = rebuild_parent_document_retriever(embeddings=embeddings,
                                                CONNECTION_STRING=CONNECTION_STRING,
                                                COLLECTION_NAME=collection_name) 
if use_colbert == True:
    print("LOADING COLBERT RERANKING MODEL")
    retriever = ContextualCompressionRetriever(
        base_compressor=reranking_model.as_langchain_document_compressor(), base_retriever=retriever
    )     
    retriever.base_compressor.k = cfg.RERANKER_VECTOR_COUNT
elif use_cross_encoder == True:
    print("LOADING CROSS ENCODER RERANKER MODEL")
    compressor = CrossEncoderReranker(model=reranking_model, top_n=cfg.RERANKER_VECTOR_COUNT)
    retriever = ContextualCompressionRetriever(
        base_compressor=compressor, base_retriever=retriever
    )       
return retriever

Error Message and Stack Trace (if applicable)

RuntimeError: The expanded size of the tensor (614) must match the existing size (512) at non-singleton dimension 1. Target sizes: [20, 614]. Tensor sizes: [1, 512]

Description

Hi,

I am using HuggingFaceCrossEncoder to rerank my retriever results for RAG. Generally it works but for some retrieval results I am getting the described error message. I checked the issue and assume that my input is too long as the max_tokens of the model "cross-encoder/msmarco-MiniLM-L12-en-de-v1" is 512.

Therefore I want to truncate the input in order to make it work, but found no solution when reading the docs.

Is this possible to do? Otherwise I have to stick to ColBERT for Reranking, there I don't see this issue.

System Info

langchain version: 0.1.17
langchain-community version: 0.0.36