Identifying backend compatibility versions

[LysandreJik]

We are currently working on identifying the backend versions with which we are compatible and with which we want to be compatible. These backends are PyTorch and TensorFlow. We will be considering Flax at a later point in time.

The first step was to identify the number of failures in each PyTorch/TensorFlow version and was done in #18181.

Total number of tests: 38,991.

Framework	No. Failures	Release date	Older than 2 years
PyTorch 1.10	50	Mar 10 2021	No
PyTorch 1.9	710	Jun 15 2021	No
PyTorch 1.8	1301	Mar 4 2021	No
PyTorch 1.7	1567	Oct 27 2020	No
PyTorch 1.6	2342	Jul 28 2020	Yes
PyTorch 1.5	3315	Apr 21 2020	Yes
PyTorch 1.4	3949	Jan 16 2020	Yes
TensorFlow 2.8	118	Feb 2 2022	No
TensorFlow 2.7	122	Nov 4 2021	No
TensorFlow 2.6	122	Aug 11 2021	No
TensorFlow 2.5	128	May 13 2021	No
TensorFlow 2.4	167	Dec 14 2020	No

We're proposing to drop versions older than 2 years old and to work towards providing support (support = 0 tests failing) for versions we aim to support. We will drop support for older versions once we reach their two-year-old date.

Here is the proposed plan moving forward:

• Have a detailed breakdown of failures for the following versions:
  • Torch 1.7
  • Torch 1.8
  • Torch 1.9
  • Torch 1.10
  • Torch 1.11
  • Torch 1.12
  • TensorFlow 2.4
  • TensorFlow 2.5
  • TensorFlow 2.6
  • TensorFlow 2.7
  • TensorFlow 2.8
  • TensorFlow 2.9
• Start with an initial compatibility document to mention which models are supported in which versions
• Open good first issues to improve compatibility for models not compatible with all versions, starting from the latest one and moving back in time.
• As versions become supported, run tests on older versions to ensure no regression.

Work by @ydshieh and @LysandreJik

---

Some context and tips when working on Past CI

1. The Past CI runs against a specific commit/tag:
   • Motivation: To be able to run the test against the same commit to see if a set of fixes improves the overall backward compatibility without new issues introduced.
   • The chosen commit could be changed (to more recent ones) along the time, but it should never be main.
   • When working on the fix for Past CI , keeping in mind that we should check the source code in the commit that is chosen for that particular Past CI run. The commit given at the beginning of each report provided in the following comments.
2. For each report, there is an attached errors.txt where you can find more information to ease the fix process:
   • The file contains a list whose elements have the following content:
     • The line where an error occurs
     • The error message
     • The complete name of the failed test
     • The link to the job that ran that failed test
   • The errors in the reports sometimes don't contain enough information to make the decision/action. You can use the corresponding links provided in errors.txt to see the full trackback on the job run pages.
3. One (possible) fix process would be like:
   • For a framework and a particular version, go to the corresponding reporting table provided in the following comments.
   • Make sure you have a preferred way to navigate the source code in a specific commit.
   • Download/Open the corresponding errors.txt.
   • From the General table, take a row whose status is empty. Ideally, take the ones with higher value in no. column.
   • Search in errors.txt for the error in the picked row. You get information about the failed line, failed test, and the job link.
   • Navigate to the failed line or failed test in your workspace (or in a browser) that checks out to the specific commit for the run.
   • Use the job link to go to the job run page if you need more information about the error.
   • Then you might come up with a solution :-), or decide a fix is not necessary with good reasons.
   • Update the status column with a comment once a fix or a decision is made.
4. Some guides/hints for the fix:
   • 🔥 To install a specific framework version, utils/past_ci_versions.py can help!
   • ⚠️ As the tests are run against a chosen commit, which may not contain some fixes in the main branch. (This is particular confusing if you try to run the failed test without checking out to that commit.).
     • If the test passes when you run a failed test (in the report) against the main branch, with the target framework version, it's very likely a fix exists on main that applies to the target framework version too.
     • In this case,
       • either update status with fixed in #XXXXX (if you know clearly that PR fixes that error)
       • or works for commits since **b487096** - a commit sha (It's not always trivial to find out which PR fixed a particular error - especially when working with Past CI)
   • We decide to focus on the PyTorch and TensorFlow version, and not to consider other 3rd libraries. Therefore, some packages are not installed, like kenlm or detectorn2. We could just simply update the status column with XXX not installed.
   • When an error is coming from a C/C++ exception, and the same code and inputs work for new framework versions, we could skip that failed test with a @unittest.skipIf, and update the status like torch._C issue -> works wth PT >= 11 Fixed in #19122.
     • PR #19122 is one such example.
   • If an error occurs in several framework versions, say, PT 11 and PT 10, and a status is updated for the newer version (here PT 11), we can simply put see PT 11 in the report status column for older versions.
   • Some old framework versions lack attributes or arguments introduced in newer versions. See #19201 and #19203 for how a fix would look like in such cases. If a similar warning (to the one in #19203) already exists, we could update status with, for example, Vilt needs PT >= 1.10.
     • Adding such warning is not a fix in a strict sense, but at least it provides some information. Together with the updated status, we keep information tracked.

[ydshieh]

Past CI - PyTorch 1.11 (Patch release: v4.21.2 | b487096)

General

no.	error	status
32	NameError: name 'kenlm' is not defined	not installed
12	RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0	fixed in #18303
6	OSError: gs555750 is not a valid git identifier (branch name, tag name or commit id) that exists for	fixed in #18531
6	NameError: name 'GPT2Tokenizer' is not defined	fixed in #19010
6	TypeError: forward() missing 1 required positional argument: 'attention_mask'	fixed in #18303
3	ImportError:	detectron2 and accelerate not installed
2	AssertionError: torch.Size([1, 2]) != torch.Size([1, 32])	fixed in #18303
1	RuntimeError: Caught RuntimeError in replica 0 on device 0.	fixed in #18303

Per model

model	no. of errors	major error	count
wav2vec2_with_lm	30	NameError: name 'kenlm' is not defined	30
owlvit	21	RuntimeError: Expected all tensors to be on the same device,	12
opt	6	NameError: name 'GPT2Tokenizer' is not defined	6
bloom	6	OSError: gs555750 is not a valid git identifier (branch name	6
wav2vec2	2	NameError: name 'kenlm' is not defined	2
layoutlmv2	2	ImportError:	2

[ydshieh]

Past CI - PyTorch 1.10 (Patch release: v4.21.2 | b487096)

General

no.	error	status
32	NameError: name 'kenlm' is not defined	see PT 11
12	RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0	see PT 11
6	OSError: gs555750 is not a valid git identifier (branch name, tag name or commit id) that exists for	see PT 11
6	NameError: name 'GPT2Tokenizer' is not defined	see PT 11
6	TypeError: forward() missing 1 required positional argument: 'attention_mask'	see PT 11
4	RuntimeError: Index is supposed to be an empty tensor or a vector	torch._C issue -> works wth PT >= 11 Fixed in #19122
3	ImportError:	see PT 11
2	AssertionError: 1.9311904907226562e-05 != 1.9431114196777344e-05	self.assertEqual is too strict. Fixed in #19200
2	AssertionError: torch.Size([1, 2]) != torch.Size([1, 32])	see PT 11
1	RuntimeError: Caught RuntimeError in replica 0 on device 0.	see PT 11

Per model

model	no. of errors	major error	count
wav2vec2_with_lm	30	NameError: name 'kenlm' is not defined	30
owlvit	21	RuntimeError: Expected all tensors to be on the same device,	12
bloom	8	OSError: gs555750 is not a valid git identifier (branch name	6
opt	6	NameError: name 'GPT2Tokenizer' is not defined	6
longt5	4	RuntimeError: Index is supposed to be an empty tensor or a v	4
wav2vec2	2	NameError: name 'kenlm' is not defined	2
layoutlmv2	2	ImportError:	2

[ydshieh]

Past CI - PyTorch 1.9 (Patch release: v4.21.2 | b487096)

errors-pt-1-9.txt

General

no.	error	status
50	AttributeError: module 'torch' has no attribute 'pi'	Need PT >= 1.10. But we can use np.pi. See #19201
44	TypeError: meshgrid() got an unexpected keyword argument 'indexing'	Vilt needs PT >= 1.10
32	NameError: name 'kenlm' is not defined	see PT 11
18	AttributeError: module 'torchaudio.functional' has no attribute 'melscale_fbanks'	Need torchaudio >= 0.10. See #19203
15	RuntimeError: CUDA error: an illegal memory access was encountered	LeViT re-run OK
12	RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0	see PT 11
6	OSError: gs555750 is not a valid git identifier (branch name, tag name or commit id) that exists for	see PT 11
6	NameError: name 'GPT2Tokenizer' is not defined	see PT 11
6	TypeError: forward() missing 1 required positional argument: 'attention_mask'	see PT 11
3	ImportError:	see PT 11
2	RuntimeError: "LayerNormKernelImpl" not implemented for 'BFloat16'	fixed in #19261
2	AssertionError: 1.9311904907226562e-05 != 1.9431114196777344e-05	See PT 10
2	AssertionError: -198.98219299316406 != -198.98225 within 4 places (5.7006835930906163e-05 difference	diff acceptable
2	RuntimeError: Index is supposed to be an empty tensor or a vector	torch._C issue -> works wth PT >= 11 Fixed in #19122
2	RuntimeError: Expected node type 'onnx::Constant' for argument 'num_classes' of node 'one_hot', got	test already skipped in #19122 (due to another error)
2	AssertionError: torch.Size([1, 2]) != torch.Size([1, 32])	see PT 11
2	TypeError: Caught TypeError in replica 0 on device 0.	Vilt needs PT >= 1.10 (meshgrid error)
1	RuntimeError: transform: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access wa	See #20859 (opened)
1	RuntimeError: Caught RuntimeError in replica 0 on device 0.	see PT 11

Per model

model	no. of errors	major error	count
maskformer	50	AttributeError: module 'torch' has no attribute 'pi'	50
vilt	46	TypeError: meshgrid() got an unexpected keyword argument 'in	44
wav2vec2_with_lm	30	NameError: name 'kenlm' is not defined	30
owlvit	21	RuntimeError: Expected all tensors to be on the same device,	12
mctct	18	AttributeError: module 'torchaudio.functional' has no attrib	18
levit	16	RuntimeError: CUDA error: an illegal memory access was encou	15
bloom	10	OSError: gs555750 is not a valid git identifier (branch name	6
opt	6	NameError: name 'GPT2Tokenizer' is not defined	6
longt5	4	RuntimeError: Index is supposed to be an empty tensor or a v	2
flava	2	AssertionError: -198.98219299316406 != -198.98225 within 4 p	2
wav2vec2	2	NameError: name 'kenlm' is not defined	2
layoutlmv2	2	ImportError:	2

[ydshieh]

Past CI - PyTorch 1.8 (Patch release: v4.21.2 | b487096)

errors-pt-1-8.txt

General

no.	error	status
570	AttributeError: module 'torch.jit._state' has no attribute '_clear_class_state'	WIP
50	AttributeError: module 'torch' has no attribute 'pi'	See PT 1.9
44	TypeError: conv1d(): argument 'padding' (position 5) must be tuple of ints, not str	WIP
44	TypeError: meshgrid() got an unexpected keyword argument 'indexing'	See PT 1.9
30	NameError: name 'kenlm' is not defined	see PT 11
26	AttributeError: module 'torch' has no attribute 'permute'	WIP
18	AttributeError: module 'torchaudio.functional' has no attribute 'melscale_fbanks'	See PT 1.9
12	RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0	see PT 1.11
8	RuntimeError: einsum() operand subscript must be in range [a, z] but found B for operand 0	WIP
6	OSError: gs555750 is not a valid git identifier (branch name, tag name or commit id) that exists for	see PT 1.11
6	NameError: name 'GPT2Tokenizer' is not defined	see PT 1.11
6	TypeError: forward() missing 1 required positional argument: 'attention_mask'	see PT 1.11
4	TypeError: Caught TypeError in replica 0 on device 0.	see PT 1.10
3	ImportError:	See PT 1.11
2	RuntimeError: "LayerNormKernelImpl" not implemented for 'BFloat16'	See PT 1.9
2	RuntimeError: "min_cuda" not implemented for 'BFloat16'	WIP
2	AssertionError: 1.9311904907226562e-05 != 1.9431114196777344e-05	See PT 10
2	AssertionError: -198.98219299316406 != -198.98225 within 4 places (5.7006835930906163e-05 difference	See PT 9
2	AssertionError: False is not true
2	RuntimeError: Expected node type 'onnx::Constant' for argument 'num_classes' of node 'one_hot', got	See PT 1.9
2	AssertionError: torch.Size([1, 2]) != torch.Size([1, 32])	see PT 11
2	TypeError: CheckpointFunctionBackward.forward: expected Tensor or tuple of Tensor (got tuple) for re	WIP
2	TypeError: save_for_backward can only save variables, but argument 2 is of type bool	WIP
2	AttributeError: module 'torchaudio.functional' has no attribute 'resample'	WIP
1	RuntimeError: Caught RuntimeError in replica 0 on device 0.	see PT 11
1	AssertionError: 2.9253265857696533 != 2.925307273864746 within 1e-05 delta (1.9311904907226562e-05 d	diff acceptable

Per model

model	no. of errors	major error	count
mctct	64	TypeError: conv1d(): argument 'padding' (position 5) must be	44
maskformer	50	AttributeError: module 'torch' has no attribute 'pi'	50
vilt	46	TypeError: meshgrid() got an unexpected keyword argument 'in	44
owlvit	33	AttributeError: module 'torch.jit._state' has no attribute '	12
wav2vec2_with_lm	30	NameError: name 'kenlm' is not defined	30
longt5	26	AttributeError: module 'torch.jit._state' has no attribute '	24
perceiver	26	AttributeError: module 'torch' has no attribute 'permute'	26
bloom	18	OSError: gs555750 is not a valid git identifier (branch name	6
prophetnet	18	AttributeError: module 'torch.jit._state' has no attribute '	18
data2vec	18	AttributeError: module 'torch.jit._state' has no attribute '	18
hubert	14	AttributeError: module 'torch.jit._state' has no attribute '	12
realm	14	RuntimeError: einsum() operand subscript must be in range [a	8
wav2vec2	14	AttributeError: module 'torch.jit._state' has no attribute '	12
clip	12	AttributeError: module 'torch.jit._state' has no attribute '	12
marian	12	AttributeError: module 'torch.jit._state' has no attribute '	12
opt	12	AttributeError: module 'torch.jit._state' has no attribute '	6
blenderbot	12	AttributeError: module 'torch.jit._state' has no attribute '	12
pegasus	12	AttributeError: module 'torch.jit._state' has no attribute '	12
t5	12	AttributeError: module 'torch.jit._state' has no attribute '	12
funnel	12	AttributeError: module 'torch.jit._state' has no attribute '	12
mvp	12	AttributeError: module 'torch.jit._state' has no attribute '	12
mbart	12	AttributeError: module 'torch.jit._state' has no attribute '	12
plbart	12	AttributeError: module 'torch.jit._state' has no attribute '	12
blenderbot_small	12	AttributeError: module 'torch.jit._state' has no attribute '	12
bart	12	AttributeError: module 'torch.jit._state' has no attribute '	12
swin	8	AttributeError: module 'torch.jit._state' has no attribute '	6
sew	6	AttributeError: module 'torch.jit._state' has no attribute '	6
resnet	6	AttributeError: module 'torch.jit._state' has no attribute '	6
xlm_roberta_xl	6	AttributeError: module 'torch.jit._state' has no attribute '	6
splinter	6	AttributeError: module 'torch.jit._state' has no attribute '	6
dpt	6	AttributeError: module 'torch.jit._state' has no attribute '	6
xlm	6	AttributeError: module 'torch.jit._state' has no attribute '	6
speech_to_text_2	6	AttributeError: module 'torch.jit._state' has no attribute '	6
dpr	6	AttributeError: module 'torch.jit._state' has no attribute '	6
squeezebert	6	AttributeError: module 'torch.jit._state' has no attribute '	6
vit	6	AttributeError: module 'torch.jit._state' has no attribute '	6
mobilebert	6	AttributeError: module 'torch.jit._state' has no attribute '	6
convnext	6	AttributeError: module 'torch.jit._state' has no attribute '	6
xlnet	6	AttributeError: module 'torch.jit._state' has no attribute '	6
glpn	6	AttributeError: module 'torch.jit._state' has no attribute '	6
segformer	6	AttributeError: module 'torch.jit._state' has no attribute '	6
cpm	6	AttributeError: module 'torch.jit._state' has no attribute '	6
nezha	6	AttributeError: module 'torch.jit._state' has no attribute '	6
bigbird_pegasus	6	AttributeError: module 'torch.jit._state' has no attribute '	6
megatron_bert	6	AttributeError: module 'torch.jit._state' has no attribute '	6
trocr	6	AttributeError: module 'torch.jit._state' has no attribute '	6
rembert	6	AttributeError: module 'torch.jit._state' has no attribute '	6
van	6	AttributeError: module 'torch.jit._state' has no attribute '	6
mobilevit	6	AttributeError: module 'torch.jit._state' has no attribute '	6
gpt_neox	6	AttributeError: module 'torch.jit._state' has no attribute '	6
openai	6	AttributeError: module 'torch.jit._state' has no attribute '	6
albert	6	AttributeError: module 'torch.jit._state' has no attribute '	6
nystromformer	6	AttributeError: module 'torch.jit._state' has no attribute '	6
distilbert	6	AttributeError: module 'torch.jit._state' has no attribute '	6
gpt2	6	AttributeError: module 'torch.jit._state' has no attribute '	6
mpnet	6	AttributeError: module 'torch.jit._state' has no attribute '	6
roberta	6	AttributeError: module 'torch.jit._state' has no attribute '	6
deit	6	AttributeError: module 'torch.jit._state' has no attribute '	6
unispeech	6	AttributeError: module 'torch.jit._state' has no attribute '	6
flaubert	6	AttributeError: module 'torch.jit._state' has no attribute '	6
codegen	6	AttributeError: module 'torch.jit._state' has no attribute '	6
wavlm	6	AttributeError: module 'torch.jit._state' has no attribute '	6
xglm	6	AttributeError: module 'torch.jit._state' has no attribute '	6
roformer	6	AttributeError: module 'torch.jit._state' has no attribute '	6
regnet	6	AttributeError: module 'torch.jit._state' has no attribute '	6
bert_generation	6	AttributeError: module 'torch.jit._state' has no attribute '	6
convbert	6	AttributeError: module 'torch.jit._state' has no attribute '	6
beit	6	AttributeError: module 'torch.jit._state' has no attribute '	6
transfo_xl	6	AttributeError: module 'torch.jit._state' has no attribute '	6
electra	6	AttributeError: module 'torch.jit._state' has no attribute '	6
ctrl	6	AttributeError: module 'torch.jit._state' has no attribute '	6
canine	6	AttributeError: module 'torch.jit._state' has no attribute '	6
groupvit	6	AttributeError: module 'torch.jit._state' has no attribute '	6
gptj	6	AttributeError: module 'torch.jit._state' has no attribute '	6
gpt_neo	6	AttributeError: module 'torch.jit._state' has no attribute '	6
fnet	6	AttributeError: module 'torch.jit._state' has no attribute '	6
fsmt	6	AttributeError: module 'torch.jit._state' has no attribute '	6
m2m_100	6	AttributeError: module 'torch.jit._state' has no attribute '	6
layoutlm	6	AttributeError: module 'torch.jit._state' has no attribute '	6
layoutlmv2	2	ImportError:	2
flava	2	AssertionError: -198.98219299316406 != -198.98225 within 4 p	2
trajectory_transformer	2	TypeError: save_for_backward can only save variables, but ar	2

[ydshieh]

Past CI - TensorFlow 2.8 (Patch release: v4.21.2 | b487096)

General

no.	error
66	RuntimeError: Cannot export model to ONNX using PyTorch because no PyTorch package was found.
30	NameError: name 'kenlm' is not defined
6	NameError: name 'GPT2Tokenizer' is not defined
4	NameError: name 'MaskFormerForInstanceSegmentation' is not defined
4	ImportError:
2	NameError: name 'MaskFormerModel' is not defined
2	tensorflow.python.framework.errors_impl.InvalidArgumentError: required broadcastable shapes [Op:Equa
1	ValueError: You called set_weights(weights) on layer "tf_segformer_for_image_classification_8" wit

Per model

model	no. of errors	major error	count
wav2vec2_with_lm	28	NameError: name 'kenlm' is not defined	28
maskformer	6	NameError: name 'MaskFormerForInstanceSegmentation' is not d	4
opt	6	NameError: name 'GPT2Tokenizer' is not defined	6
speech_to_text	4	ImportError:	4
wav2vec2	2	NameError: name 'kenlm' is not defined	2
rembert	2	tensorflow.python.framework.errors_impl.InvalidArgumentError	2
segformer	1	ValueError: You called set_weights(weights) on layer "tf_s	1

[ydshieh]

Past CI - TensorFlow 2.7 (Patch release: v4.21.2 | b487096)

General

no.	error
66	RuntimeError: Cannot export model to ONNX using PyTorch because no PyTorch package was found.
30	NameError: name 'kenlm' is not defined
6	TypeError: Invalid keyword argument(s) in compile(): ({'jit_compile'},). Valid keyword arguments i
6	NameError: name 'GPT2Tokenizer' is not defined
4	NameError: name 'MaskFormerForInstanceSegmentation' is not defined
4	ImportError:
2	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
2	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
2	NameError: name 'MaskFormerModel' is not defined
2	tensorflow.python.framework.errors_impl.InvalidArgumentError: required broadcastable shapes [Op:Equa
1	ValueError: You called set_weights(weights) on layer "tf_segformer_for_image_classification_8" wit
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t

Per model

model	no. of errors	major error	count
wav2vec2_with_lm	28	NameError: name 'kenlm' is not defined	28
t5	10	tensorflow.python.framework.errors_impl.InvalidArgumentError	1
maskformer	6	NameError: name 'MaskFormerForInstanceSegmentation' is not d	4
opt	6	NameError: name 'GPT2Tokenizer' is not defined	6
speech_to_text	4	ImportError:	4
wav2vec2	2	NameError: name 'kenlm' is not defined	2
gptj	2	TypeError: Invalid keyword argument(s) in compile(): ({'ji	2
bart	2	TypeError: Invalid keyword argument(s) in compile(): ({'ji	2
gpt2	2	TypeError: Invalid keyword argument(s) in compile(): ({'ji	2
rembert	2	tensorflow.python.framework.errors_impl.InvalidArgumentError	2
segformer	1	ValueError: You called set_weights(weights) on layer "tf_s	1

[ydshieh]

Past CI - TensorFlow 2.6 (Patch release: v4.21.2 | b487096)

General

no.	error
66	RuntimeError: Cannot export model to ONNX using PyTorch because no PyTorch package was found.
30	NameError: name 'kenlm' is not defined
10	ValueError: in user code:
6	TypeError: Invalid keyword argument(s) in compile: {'jit_compile'}
6	NameError: name 'GPT2Tokenizer' is not defined
4	NameError: name 'MaskFormerForInstanceSegmentation' is not defined
4	ImportError:
2	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
2	tensorflow.python.framework.errors_impl.InvalidArgumentError: Detected unsupported operations when t
2	NameError: name 'MaskFormerModel' is not defined
2	tensorflow.python.framework.errors_impl.InvalidArgumentError: required broadcastable shapes [Op:Equa
1	ValueError: You called set_weights(weights) on layer "tf_segformer_for_image_classification_8" wit
1	ValueError: Unable to save function b'__inference_tf_speech2text_model_25_layer_call_and_return_cond
1	tensorflow.python.framework.errors_impl.InvalidArgumentError: Expected 'tf.Tensor(False, shape=(), d
1	ValueError: Unable to save function b'__inference_tf_speech2text_model_25_layer_call_and_return_cond

Per model

model	no. of errors	major error	count
wav2vec2_with_lm	28	NameError: name 'kenlm' is not defined	28
t5	10	ValueError: in user code:	10
maskformer	6	NameError: name 'MaskFormerForInstanceSegmentation' is not d	4
opt	6	NameError: name 'GPT2Tokenizer' is not defined	6
speech_to_text	6	ImportError:	4
bart	3	TypeError: Invalid keyword argument(s) in compile: {'jit_c	2
wav2vec2	2	NameError: name 'kenlm' is not defined	2
gptj	2	TypeError: Invalid keyword argument(s) in compile: {'jit_c	2
gpt2	2	TypeError: Invalid keyword argument(s) in compile: {'jit_c	2
rembert	2	tensorflow.python.framework.errors_impl.InvalidArgumentError	2
segformer	1	ValueError: You called set_weights(weights) on layer "tf_s	1

[ydshieh]

Past CI - TensorFlow 2.5 (Patch release: v4.21.2 | b487096)

General

no.	error
70	RuntimeError: Failed to import transformers.models.albert.modeling_tf_albert because of the followin
28	NameError: name 'kenlm' is not defined
18	RuntimeError: Failed to import transformers.models.gpt2.modeling_tf_gpt2 because of the following er
4	NameError: name 'MaskFormerForInstanceSegmentation' is not defined
2	RuntimeError: Failed to import transformers.models.t5.modeling_tf_t5 because of the following error
2	RuntimeError: Failed to import transformers.models.distilbert.modeling_tf_distilbert because of the
2	RuntimeError: Failed to import transformers.models.bert.modeling_tf_bert because of the following er
2	NameError: name 'MaskFormerModel' is not defined

Per model

model	no. of errors	major error	count
wav2vec2_with_lm	28	NameError: name 'kenlm' is not defined	28
maskformer	6	NameError: name 'MaskFormerForInstanceSegmentation' is not d	4
squeezebert	4	RuntimeError: Failed to import transformers.models.albert.mo	4
xglm	2	RuntimeError: Failed to import transformers.models.albert.mo	2
bert_generation	2	RuntimeError: Failed to import transformers.models.albert.mo	2
byt5	2	RuntimeError: Failed to import transformers.models.albert.mo	2
bloom	2	RuntimeError: Failed to import transformers.models.albert.mo	2
perceiver	2	RuntimeError: Failed to import transformers.models.albert.mo	2
layoutlmv2	2	RuntimeError: Failed to import transformers.models.albert.mo	2
bort	2	RuntimeError: Failed to import transformers.models.bert.mode	2
tapex	2	RuntimeError: Failed to import transformers.models.albert.mo	2
plbart	2	RuntimeError: Failed to import transformers.models.albert.mo	2
barthez	2	RuntimeError: Failed to import transformers.models.albert.mo	2
layoutxlm	2	RuntimeError: Failed to import transformers.models.albert.mo	2
nllb	2	RuntimeError: Failed to import transformers.models.albert.mo	2
canine	2	RuntimeError: Failed to import transformers.models.albert.mo	2
layoutlmv3	2	RuntimeError: Failed to import transformers.models.albert.mo	2
xlm_prophetnet	2	RuntimeError: Failed to import transformers.models.albert.mo	2
luke	2	RuntimeError: Failed to import transformers.models.albert.mo	2
mbart50	2	RuntimeError: Failed to import transformers.models.albert.mo	2
realm	2	RuntimeError: Failed to import transformers.models.albert.mo	2
mluke	2	RuntimeError: Failed to import transformers.models.albert.mo	2
bertweet	2	RuntimeError: Failed to import transformers.models.albert.mo	2
mvp	2	RuntimeError: Failed to import transformers.models.albert.mo	2
big_bird	2	RuntimeError: Failed to import transformers.models.albert.mo	2
phobert	2	RuntimeError: Failed to import transformers.models.albert.mo	2
fnet	2	RuntimeError: Failed to import transformers.models.albert.mo	2
speech_to_text_2	2	RuntimeError: Failed to import transformers.models.albert.mo	2
prophetnet	2	RuntimeError: Failed to import transformers.models.albert.mo	2
herbert	2	RuntimeError: Failed to import transformers.models.albert.mo	2
fsmt	2	RuntimeError: Failed to import transformers.models.albert.mo	2
codegen	2	RuntimeError: Failed to import transformers.models.albert.mo	2
retribert	2	RuntimeError: Failed to import transformers.models.albert.mo	2
m2m_100	2	RuntimeError: Failed to import transformers.models.albert.mo	2
bartpho	2	RuntimeError: Failed to import transformers.models.albert.mo	2
reformer	2	RuntimeError: Failed to import transformers.models.albert.mo	2

[LysandreJik]

I was trying to fix the kenlm issue, but I see it's correctly installed here and has been for a while.

I guess it is an image issue?

[ydshieh]

I was trying to fix the kenlm issue, but I see it's correctly installed here and has been for a while.

I guess it is an image issue?

Hi @LysandreJik. In fact, Past CI use transformers-past-gpu/Dockerfile:
https://github.com/huggingface/transformers/blame/main/docker/transformers-past-gpu/Dockerfile

It's probably arguable if we should (or should not) include kenlm. I don't remember well if I got issue when installing it. Maybe yes for more elder versions, so I decide not to install it for all versions (to avoid confusion).

We can try with it in the next launch.

[LysandreJik]

I think we can add it, we've had it in the main file for 8 months so it's unlikely to cause an issue. Looking forward to the next launch!