Mixture of non-prefixed and prefixed (B-, I-)

[KoichiYasuoka]

What does this PR do?

Reconsider of #13493 for mixture case, where B-TAG and I-TAG are used for multi-tokens and TAG is used for a single token.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@Narsil

[Narsil]

This seems a bit overboard. I don't think mixed tags is desirable. (it's very uneasy to know what a model will do.
The previous fix was about being backward compatible, this is a proposed breaking change I think.

However I am not necessarily having all information.

@LysandreJik do you have an opinion ?

[LysandreJik]

Pinging @stefan-it as he has extensive experience with NER, maybe you can enlighten us? :)

[stefan-it]

Yeah, I think this is also related to this issue:

#4262

tldr: some of the NER datasets use IOB1 as labeling scheme (namely CoNLL-2003 English and German).

[Narsil]

Hi @stefan-it , but reading the conversation it just confirms that using I- only is the way to go, and that we cannot have same but separated entity splits with this scheme.

It seems that the proposed PR wouldn't actually solve the problem either because B- are never outputted, right ?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.