Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(utils): remove non-ASCII limitation for path normalization #8137

Merged
merged 4 commits into from Oct 7, 2022

Conversation

birjj
Copy link
Contributor

@birjj birjj commented Sep 27, 2022

Pre-flight checklist

  • I have read the Contributing Guidelines on pull requests.
  • If this is a code change: I have written unit tests and/or added dogfooding pages to fully verify the new behavior.
  • If this is a new API or substantial change: the PR has an accompanying issue (closes #0000) and the maintainers have approved on my working plan.

Motivation

Fixes #8124

The issue specifically pertains to index file detection for paths that contain non-ASCII letters (e.g. "æ"). This failed previously, due to a test for non-ASCII characters when normalizing file paths. This test originated in the sindresorhus/slash package, and was copied over when the code was integrated into docusaurus - but doesn't appear to match how Windows works.

See that issue for further discussion, or sindresorhus/slash#19 for discussion on limitation in the originating package. The limitation was removed in the originating package in sindresorhus/slash#20

Test Plan

Updated tests in this repo, and verified by modifying node_modules directly in https://github.com/birjj/docusaurus-repro-8124 that it works for the specific issue encountered:

Image comparing a Docusaurus sidebar before and after the change, show an expandable

For the file structure

docs
 ├─ abc
 │  └─ index.md
 └─ æøå
    └─ index.md

Test links

Deploy preview: https://deploy-preview-8137--docusaurus-2.netlify.app/ (note that this isn't relevant, as the issue is Windows-specific)

Related issues/PRs

#8124
sindresorhus/slash#19
sindresorhus/slash#20

See sindresorhus/slash#19 for discussion on limitation in the originating package
This updates the integrated function to match sindresorhus/slash after sindresorhus/slash#20
@facebook-github-bot
Copy link
Contributor

Hi @birjj!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

@netlify
Copy link

netlify bot commented Sep 27, 2022

[V2]

Name Link
🔨 Latest commit 70d31a3
🔍 Latest deploy log https://app.netlify.com/sites/docusaurus-2/deploys/63400b333b0c0000092d1f6d
😎 Deploy Preview https://deploy-preview-8137--docusaurus-2.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions
Copy link

github-actions bot commented Sep 27, 2022

⚡️ Lighthouse report for the deploy preview of this PR

URL Performance Accessibility Best Practices SEO PWA Report
/ 🟠 75 🟢 98 🟢 100 🟢 100 🟠 80 Report
/docs/installation 🟠 73 🟢 100 🟢 100 🟢 100 🟢 90 Report

@facebook-github-bot
Copy link
Contributor

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

@facebook-github-bot facebook-github-bot added the CLA Signed Signed Facebook CLA label Sep 27, 2022
@birjj birjj changed the title fix: remove non-ASCII limitation for path normalization fix(utils): remove non-ASCII limitation for path normalization Sep 27, 2022
@Josh-Cena
Copy link
Collaborator

There's another escapePath utility we use to get around the issue. I think we can merge the two now.

@slorber
Copy link
Collaborator

slorber commented Sep 29, 2022

Also questioned this part of the code before, see some past refs here:

Was wondering: is there a way to prevent this issue to resurface again with CI? Any idea to share? (we have Wndows CI but no e2e tests infra yet)

We have a website/_dogfood folder to have tests, is there a way to make the Docusaurus website build crash just using some md files + windows?


There's another escapePath utility we use to get around the issue. I think we can merge the two now.

The implementation looks quite different, if we want to dedup I'd rather do that in another dedicated PR as this looks more risky 😰

/**
 * When you have a path like C:\X\Y
 * It is not safe to use directly when generating code
 * For example, this would fail due to unescaped \:
 * `<img src={require('${filePath}')} />`
 * But this would work: `<img src={require('${escapePath(filePath)}')} />`
 *
 * posixPath can't be used in all cases, because forward slashes are only valid
 * Windows paths when they don't contain non-ascii characters, and posixPath
 * doesn't escape those that fail to be converted.
 */
export function escapePath(str: string): string {
  const escaped = JSON.stringify(str);

  // Remove the " around the json string;
  return escaped.substring(1, escaped.length - 1);
}

@slorber slorber added the pr: bug fix This PR fixes a bug in a past release. label Sep 29, 2022
@birjj
Copy link
Contributor Author

birjj commented Oct 1, 2022

I agree that escapePath is probably best handled in a separate PR - and by someone more familiar with the codebase ;)

Was wondering: is there a way to prevent this issue to resurface again with CI? Any idea to share? (we have Wndows CI but no e2e tests infra yet)

We have a website/_dogfood folder to have tests, is there a way to make the Docusaurus website build crash just using some md files + windows?

I don't believe there is a way to make it crash on build, unless there's some specific edge case handling of paths that crash if they aren't converted correctly - in which case that behavior probably wouldn't be good to rely on anyway. Since slugs and document IDs aren't affected, I doubt there is any intended check that would fail.

If there is a way to test site functionality, we could test whether the index is detected correctly (#8124). Unfortunately, as far as I can see, that isn't possible with the dogfood system.

@slorber
Copy link
Collaborator

slorber commented Oct 7, 2022

Can't find a way to test it locally (I don't have windows) nor to make it fail=>pass in the CI so I'll have to trust it works 🤪

Thanks 👍

@slorber slorber added the to backport This PR is planned to be backported to a stable version of Docusaurus label Oct 7, 2022
@slorber slorber merged commit 6c8af03 into facebook:main Oct 7, 2022
@slorber slorber added backported This PR has been backported to a stable version of Docusaurus and removed to backport This PR is planned to be backported to a stable version of Docusaurus labels Nov 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backported This PR has been backported to a stable version of Docusaurus CLA Signed Signed Facebook CLA pr: bug fix This PR fixes a bug in a past release.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Autogenerated sidebar fails to parse category indeces when categories have non-latin characters
4 participants