You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A bug happened!
When launching the "generate_testset" after defining the bedrock Client forClaud03, I get an error saying: "MainThread giskard.rag ERROR Encountered errorin question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-14 08:10:06,421 pid:12762 MainThread giskard.rag ERROR An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row"And just under it a "KeyError: "None of ['id'] are in the columns""PS: I don't use giskard hub
Standalone code OR list down the steps to reproduce the issue
2024-05-07 08:27:45,145 pid:12138 MainThread giskard.rag INFO Finding topics in the knowledge base.
2024-05-07 08:27:53,602 pid:12138 MainThread giskard.rag INFO Found 1 topics in the knowledge base.
Generating questions: 0%|| 0/3 [00:00<?, ?it/s]
2024-05-07 08:27:54,321 pid:12138 MainThread giskard.rag ERROR Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-07 08:27:54,330 pid:12138 MainThread giskard.rag ERROR An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 57, in generate_questions
yield self.generate_single_question(knowledge_base, *args, **kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/simple_questions.py", line 96, in generate_single_question
generated_qa = self._llm_complete(messages=messages)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 42, in _llm_complete
out = self._llm_client.complete(
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/llm/client/bedrock.py", line 72, incomplete
response = self._client.invoke_model(body=body, modelId=self.model, accept=accept, contentType=contentType)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
2024-05-07 08:27:54,530 pid:12138 MainThread giskard.rag ERROR Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-07 08:27:54,538 pid:12138 MainThread giskard.rag ERROR An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 57, in generate_questions
yield self.generate_single_question(knowledge_base, *args, **kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/simple_questions.py", line 96, in generate_single_question
generated_qa = self._llm_complete(messages=messages)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 42, in _llm_complete
out = self._llm_client.complete(
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/llm/client/bedrock.py", line 72, incomplete
response = self._client.invoke_model(body=body, modelId=self.model, accept=accept, contentType=contentType)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
2024-05-07 08:27:54,722 pid:12138 MainThread giskard.rag ERROR Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-07 08:27:54,730 pid:12138 MainThread giskard.rag ERROR An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 57, in generate_questions
yield self.generate_single_question(knowledge_base, *args, **kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/simple_questions.py", line 96, in generate_single_question
generated_qa = self._llm_complete(messages=messages)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 42, in _llm_complete
out = self._llm_client.complete(
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/llm/client/bedrock.py", line 72, incomplete
response = self._client.invoke_model(body=body, modelId=self.model, accept=accept, contentType=contentType)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/tmp/ipykernel_12138/3725230573.py in?()
----> 1 testset = generate_testset(knowledge_base,
2 num_questions=tuning_parameters["number_questions"],
3 agent_description="TEST")
~/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/testset_generation.py in?(knowledge_base, num_questions, question_generators, language, agent_description)
97 "question_generators": [qg.__class__.__name__ forqgin question_generators],
98 "knowledge_base_size": len(knowledge_base._documents),
99 },
100 )
--> 101 return QATestset(questions)
~/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/testset.py in?(self, question)
43 def __init__(self, question: Sequence[QuestionSample]):
44 self._questions = question
---> 45 self._dataframe = pd.DataFrame.from_records([question.to_dict() forquestionin self._questions]).set_index(
46 "id"
47 )
~/anaconda3/envs/python3/lib/python3.10/site-packages/pandas/core/frame.py in?(self, keys, drop, append, inplace, verify_integrity)
6105 if not found:
6106 missing.append(col)
6107
6108 if missing:
-> 6109 raise KeyError(f"None of {missing} are in the columns")
6110
6111 if inplace:
6112 frame = self
KeyError: "None of ['id'] are in the columns"
The text was updated successfully, but these errors were encountered:
Issue Type
Bug
Source
source
Giskard Library Version
2.11.0
Giskard Hub Version
1.3.0
OS Platform and Distribution
Linux 5.10.215-203.850.amzn2.x86_64 x86_64
Python version
3.10.14
Installed python packages
Current Behaviour?
Standalone code OR list down the steps to reproduce the issue
Relevant log output
The text was updated successfully, but these errors were encountered: