WARNING reference target not found, even though reference exists

[epwalsh]

Describe the bug

context
After upgrading MyST-Parser to the latest version, we're seeing these warnings while building our docs with Sphinx:

/home/runner/work/tango/tango/docs/source/CONTRIBUTING.md:154: WARNING: 'myst' reference target not found: #writing-docstrings
/home/runner/work/tango/tango/docs/source/CONTRIBUTING.md:179: WARNING: 'myst' reference target not found: #making-a-pull-request

MyST-Parser is complaining about internal header links in this file, such as on this line, even though those links clearly exists.

expectation
No warnings.

problem
This is causing a problem in our docs build, since we treat warnings as errors. We could silence these specific warnings, but I'd rather not do that since it could cause us to miss true positives in the future.

Reproduce the bug

git clone https://github.com/allenai/tango.git
cd tango
git checkout dependabot/pip/myst-parser-0.17.0
pip install -e '.[dev,all]'
make docs

List your environment

Python 3.9

absl-py==0.15.0
-e git+ssh://git@github.com/allenai/tango.git@f46379b9fac6cb77a2a5f8ef34a1fb208203d587#egg=ai2_tango
aiohttp==3.7.4.post0
alabaster==0.7.12
appnope==0.1.2
async-timeout==3.0.1
attrs==21.2.0
Babel==2.9.1
backcall==0.2.0
base58==2.1.0
beautifulsoup4==4.10.0
black==21.12b0
bleach==4.1.0
boto3==1.18.47
botocore==1.21.47
cached-path==1.0.2
cachetools==4.2.2
certifi==2021.5.30
chardet==4.0.0
charset-normalizer==2.0.6
click==8.0.3
click-help-colors==0.9.1
codecov==2.1.12
colorama==0.4.4
configparser==5.0.2
coverage==5.5
datasets==1.18.1
decorator==5.1.0
deepspeed==0.5.5
dill==0.3.4
distlib==0.3.4
docker-pycreds==0.4.0
docopt==0.6.2
docutils==0.17.1
fairscale==0.4.5
filelock==3.4.0
flake8==3.9.2
flaky==3.7.0
Flask==2.0.2
fsspec==2021.9.0
furo==2022.1.2
future==0.18.2
gitdb==4.0.7
GitPython==3.1.24
glob2==0.7
google-api-core==2.0.1
google-auth==2.1.0
google-auth-oauthlib==0.4.6
google-cloud-core==2.0.0
google-cloud-storage==1.42.2
google-crc32c==1.2.0
google-resumable-media==2.0.3
googleapis-common-protos==1.53.0
GPUtil==1.4.0
greenlet==1.1.1
grip==4.5.2
grpcio==1.41.1
huggingface-hub==0.2.1
idna==3.2
imagesize==1.2.0
importlib-metadata==4.8.1
iniconfig==1.1.1
ipython==7.28.0
isort==5.10.1
itsdangerous==2.0.1
jedi==0.18.0
Jinja2==3.0.1
jmespath==0.10.0
joblib==1.1.0
jsonnet==0.17.0
keyring==23.2.1
livereload==2.6.3
m2r==0.2.1
Markdown==3.3.4
markdown-it-py==1.1.0
MarkupSafe==2.0.1
matplotlib-inline==0.1.3
mccabe==0.6.1
mdit-py-plugins==0.3.0
mistune==0.8.4
more-itertools==8.10.0
msgpack==0.5.6
multidict==5.1.0
multiprocess==0.70.12.2
mypy==0.931
mypy-extensions==0.4.3
myst-parser==0.17.0
neovim==0.2.6
ninja==1.10.2.2
nltk==3.6.7
numpy==1.21.2
oauthlib==3.1.1
overrides==6.1.0
packaging==21.0
pandas==1.3.3
parso==0.8.2
path-and-address==2.0.1
pathspec==0.9.0
pathtools==0.1.2
petname==2.6
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.0.0
pkginfo==1.7.1
platformdirs==2.3.0
pluggy==1.0.0
promise==2.3
prompt-toolkit==3.0.20
protobuf==3.18.0
psutil==5.8.0
ptyprocess==0.7.0
py==1.10.0
pyarrow==5.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.7.0
pyDeprecate==0.3.1
pydocstyle==6.1.1
pyflakes==2.3.1
Pygments==2.10.0
pyparsing==2.4.7
pytest==6.2.5
pytest-cov==2.12.1
pytest-sphinx==0.3.1
python-dateutil==2.8.2
pytorch-lightning==1.5.9
pytz==2021.1
PyYAML==5.4.1
readme-renderer==29.0
regex==2021.8.28
requests==2.26.0
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
rfc3986==1.5.0
rouge-score==0.0.4
rsa==4.7.2
s3transfer==0.5.0
sacremoses==0.0.46
sentencepiece==0.1.96
sentry-sdk==1.4.3
shortuuid==1.0.1
six==1.16.0
smmap==4.0.0
snowballstemmer==2.1.0
soupsieve==2.2.1
Sphinx==4.4.0
sphinx-autobuild==2021.3.14
sphinx-copybutton==0.5.0
sphinx-design==0.0.13
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
sqlitedict==1.7.0
subprocess32==3.5.4
tensorboard==2.7.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorboardX==1.8
termcolor==1.1.0
tokenizers==0.10.3
toml==0.10.2
tomli==1.2.1
torch==1.10.1
torchaudio==0.10.1
torchmetrics==0.6.0
torchvision==0.11.2
tornado==6.1
tqdm==4.62.3
traitlets==5.1.0
transformers==4.15.0
twine==3.4.2
types-PyYAML==6.0.0
types-setuptools==57.4.2
typing-extensions==3.10.0.2
typing-utils==0.1.0
urllib3==1.26.7
virtualenv==20.13.0
wandb==0.12.4
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==2.0.2
xxhash==2.0.2
yarl==1.6.3
yaspin==2.1.0
zipp==3.5.0

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[chrisjsewell]

Heya, yes this is expected behaviour.
Before, it was treating these links starting # as an external link, but now you should explicitly use: https://myst-parser.readthedocs.io/en/latest/syntax/optional.html?highlight=anchor#auto-generated-header-anchors

Can you give that a go and let me know if it works, e.g. adding myst_heading_anchors = 1

If so I'll update the changelog, to specifically mention this change

[eyllanesc]

@chrisjsewell @epwalsh

I was testing that bug and check that with myst_heading_anchors = 3 it works (with values from 3 to 7 too).

I think that by default myst_heading_anchors should have the highest possible value or allow to use -1 to indicate it since for example for this case with 2 it does not work since one of the references is h3 and use the myst-anchors tool I think it annoying, you should only call it if placing a small myst_heading_anchors optimizes some task (if it does).

[epwalsh]

Thanks @chrisjsewell, that worked.

[andersk]

This has invalidated our workaround for #411. We were using foo.html#bar links because foo.md#bar gave incorrect “reference target not found” errors when building in parallel, but now foo.html#bar give those too.

[relsqui]

FYI, this broke our builds too for the same reason, which was an unpleasant surprise in a minor version update. I wish the changelog had been more explicit that links which did not previously raise warnings would now raise warnings unless we made a configuration change.

[chrisjsewell]

@eyllanesc myst_heading_anchors will not be enabled by default, this is because (a) it could lead to large amounts of unnecessary reference targets being generated, for people not using this feature, potentially leading to unwelcome reference clashes, and (b) there is no recognised standard naming convention for these "title slugs", docutils, github and gitlab, etc all convert title text to slugs slightly differently, hence the heading_slug_func option

@andersk I'm happy to help fix the issue, but the change will not be reversed, since it was not behaviour that was consistent across output formats

@relsqui as mentioned above, I will be updating the changelog.
It is EBP policy to treat minor version changes of packages, whose major version is 0 (i.e. 0.minor.patch), as breaking changes. This is a common practice when using semantic versioning, before packages become fully stable in 1.0.
You should expect the same from 0.18.0

[relsqui]

That's fair, re pre-1.0 versions. And I wasn't sure if the version of the changelog I'd seen was the updated one or not. :) Thanks!

[chrisjsewell]

That's fair, re pre-1.0 versions. And I wasn't sure if the version of the changelog I'd seen was the updated one or not. :) Thanks!

No worries 👍 I always try to add deprecation pathways, for any breaking changes, unfortunately in this instance I did not see any obvious way to achieve that

[JulianGro]

Same/similar problem here. It appears to be impossible now to link to RST files from MD file inside of Sphinx.
It doesn't matter which one of the following paths I give it, It always throws said warning.

entity-appearance.html#set-the-size-of-an-entity
entity-appearance#set-the-size-of-an-entity
entity-appearance.rst#set-the-size-of-an-entity
entity-appearance.md#set-the-size-of-an-entity

I guess I will have to downgrade myst-parser for now.

[chrisjsewell]

@JulianGro if entity-appearance.md#set-the-size-of-an-entity is not working, then you have not added myst_heading_anchors (or used the myst-anchors CLI to see what the links actually are)

[JulianGro]

@chrisjsewell myst_heading_anchors is set to 7 and the file isn't a markdown file but an RST file. The anchors shouldn't have changed via a myst-parser upgrade since they are not created by myst-parser.

[chrisjsewell]

the file isn't a markdown file but an RST file

Ah but that's why I only referenced entity-appearance.md#set-the-size-of-an-entity, which you said you had also tried.

If you only wish to use external html links, which is what was happening before with #, then you should use myst_all_links_external = True

[JulianGro]

Setting myst_all_links_external = True doesn't throw a warning anymore, but links to RST documentation are broken.
For example, it makes [Add a cube entity](create-entities) build as a link to create-entities instead of create-entities.html.

[scotboyd]

I'm working on a submittable repro and this may be a separate issue, but I get this warning whenever I specify -j on Sphinx (multiproc). The headers are in the same file that contains the link (usually at the top of the file) and it generates the same warning. Single proc works fine.

[andersk]

@scotboyd That’s #411, and it was fixed in #525, but there has not yet been a release including this fix.

[scotboyd]

@andersk Oh, fantastic, my google fu clearly failed me.

[flying-sheep]

Can we get myst_heading_anchors = True recognized for people who don’t want to tweak this setting?

Also the error message should mention this setting so people don’t have to find their way to this issue report

[zenichev]

Same issue by us. We are using the MD.

doing a cross-reference like this:

[some description](#link1)

Anchor looks like this:

### <a name="link1"></a> Some title here ###

the content of the conf.py has:

myst_heading_anchors = 3

In the end, during the make html I constantly get this:

<our_path>/docs/architecture.md:151: WARNING: 'myst' reference target not found: #link1

Any hints are appreciated.

[MaurycyWojda]

@zenichev I am experiencing the same issue with invisible anchors and an unstable workaround would be to make it an external link by using <a href="<path prefix since domain>/docs/architecture.html#link1">some description</a> instead of the markdown syntax.

However, since you in particular want to link to a heading, you might just want to use [some description](#some-title-here).

Nonetheless, this does not work without a heading, for example with an invisible anchor arbitrarily put in the page and I would really like it to work.

[chrisjsewell]

I'm going to close this, because the original issue was fixed: #519 (comment), plus also links have changed a bit since v0.19: https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html

Feel free to open another issue for any other problems 😄