Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cue/load: Loader does not ignore irregular files #1672

Open
kcburge opened this issue Apr 27, 2022 · 9 comments · May be fixed by #1674
Open

cue/load: Loader does not ignore irregular files #1672

kcburge opened this issue Apr 27, 2022 · 9 comments · May be fixed by #1674
Labels
NeedsInvestigation

Comments

@kcburge
Copy link
Contributor

kcburge commented Apr 27, 2022 •
edited

What version of CUE are you using (cue version)?

master , HEAD

Does this issue reproduce with the latest release?

YES

What did you do?

Using Emacs to edit my cue files, running cue fmt on save, I kept getting a file didn't exist error. (Emacs saves a symlink in the working directory to a non-existing file, as a locking mechanism). This was causing cue fmt to fail with "no such file or directory".

It could also be argued that cue should ignore files that are hidden. This symlink was a hidden file, but cue attempted to read it.

What did you expect to see?

I expect that the cue loader would:

  • ignore symlinks and other irregular files
  • ignore hidden files and directories

What did you see instead?

cue failed loading a symlink to a file that did not exist.

Test

cat <<EOF | testscript -
exec ln -s not-found .link.cue
exec cue fmt in.cue
-- cue.mod/module.cue --
module: "example.org/test"
-- sub/sub.cue --
package sub
-- in.cue --
import "example.org/test/sub"
EOF

exec ln -s not-found .link.cue
exec cue fmt in.cue
[stderr]
import failed: load: open $WORK/.link.cue: no such file or directory:
./in.cue:3:8
[exit status 1]
FAIL: /home/kevin/.tmp/testscript3413231860/-/script.txt:2: unexpected command failure
error running in /home/kevin/.tmp/testscript3413231860/-
@kcburge kcburge added NeedsInvestigation Triage Requires triage/attention labels Apr 27, 2022
kcburge added a commit to kcburge/cue that referenced this issue Apr 27, 2022
@kcburge
ignore irregular files
0c8aef0 
Load should skip symlinks and other irregular files.

closes cue-lang#1672
@kcburge kcburge mentioned this issue Apr 27, 2022
Closed
@verdverm
Copy link

verdverm commented Apr 27, 2022 •
edited

I'm curious if this change would impact those who use symlinks in cue.mod/pkg/... to reference other modules while co-developing?

would it be better to setup emacs to only format the current file being saved? (noticed that it is a single file in the repro)

I wonder if the changes to how cue fmt operates will solve this (some split so there is less loading, can't find the issue)

@kcburge
Copy link
Contributor Author

kcburge commented Apr 27, 2022

In my testing, it appears the package loading behavior for cue.mod/pkg/... is different than the module loading behavior for the current module. For example, I don't think this symlink issue occurs if I import a package with a non-existing symlink to a file. I have tried so many variations attempting to find out exactly how to reproduce this error, I've lost track.

My initial solution was to ignore a symlink to a file that did not exist. How to handle symlinks is certainly a worthy topic of discussion. But, other irregular files most certainly should be ignored.

@kcburge
Copy link
Contributor Author

kcburge commented Apr 27, 2022 •
edited

I was somewhat surprised the cue fmt caused the entire module to load. I don't know whether this is justified or not. I also believe this error will happen for eval as well. So, it's not just a fmt issue. Confirmed. Fails on eval also.

@kcburge
Copy link
Contributor Author

kcburge commented Apr 27, 2022 •
edited

For what it's worth, I know that cue complains about hidden files in some cases. The behavior with this part of the loader however is not consistent. It will gladly reference hidden files, so, actually ignoring hidden files would also be a proper solution to my problem. I would agree that a well-behaved *nix application would honor symlinks, even if I wouldn't use them here. So, perhaps the real issue is that hidden files are not ignored here.

cat <<EOF | testscript -
exec cue fmt ./...
-- cue.mod/module.cue --
module: "example.org/test"
-- sub/sub.cue --
package sub
-- in.cue --
import "example.org/test"
-- .invalid.cue --
p
EOF

exec cue fmt ./...
[stderr]
import failed: build constraints exclude all CUE files in example.org/test:
.invalid.cue: filename starts with a '.'
in.cue: no package name:
./in.cue:1:8
[exit status 1]
FAIL: /home/kevin/.tmp/testscript774736140/-/script.txt:1: unexpected command failure
error running in /home/kevin/.tmp/testscript774736140/-

I believe the above mention of "filename starts with a '.'" is just a warning.

kcburge added a commit to kcburge/cue that referenced this issue Apr 27, 2022
@kcburge
cue: ignore bad symlinks and unusable file types
8d4de61 
- Ignore symlinks whose targets cannot be stat'd
- Ignore sockets, devices, etc.

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

closes cue-lang#1672

Signed-off-by: Kevin Burge <kcburge@pm.me>
@kcburge kcburge linked a pull request Apr 27, 2022 that will close this issue
Open
@kcburge
Copy link
Contributor Author

kcburge commented Apr 27, 2022

submitted an alternate implementation that preserves symlink capability

kcburge added a commit to kcburge/cue that referenced this issue Apr 28, 2022
@kcburge
cue: ignore bad symlinks and unusable file types
59ed280 
- Ignore symlinks whose targets cannot be stat'd
- Ignore sockets, devices, etc.

closes cue-lang#1672

Signed-off-by: Kevin Burge <kcburge@pm.me>
kcburge added a commit to kcburge/cue that referenced this issue Apr 28, 2022
@kcburge
cue: ignore bad symlinks and unusable file types
16fc1d2 
- Ignore missing symlinks,
- Fail on symlinks that cannot be stat'd
- Only process directories and regular files

closes cue-lang#1672

Signed-off-by: Kevin Burge <kcburge@pm.me>
@kcburge
Copy link
Contributor Author

kcburge commented Apr 28, 2022

I believe the correct behavior is to only consider files and directories and symlinks to files and directories that exist. If a symlink cannot be stat'd, an error should return, just like directories that cannot be read. Updated my pull request to do this.

kcburge added a commit to kcburge/cue that referenced this issue Apr 28, 2022
@kcburge
cue: ignore bad symlinks and unusable file types
2b449ca 
- Ignore missing symlinks,
- Fail on symlinks that cannot be stat'd
- Only process directories and regular files

closes cue-lang#1672

Signed-off-by: Kevin Burge <kcburge@pm.me>
@rogpeppe rogpeppe removed the Triage Requires triage/attention label Apr 28, 2022
@myitcv
Copy link
Member

myitcv commented Apr 29, 2022

I'm not clear why .link.cue is being walked in the first place. Like Go, files and directories that begin with _ and . should be ignored.

@myitcv myitcv changed the title Loader does not ignore irregular files cue/load: Loader does not ignore irregular files Apr 29, 2022
@myitcv
Copy link
Member

myitcv commented Apr 29, 2022 •
edited

@kcburge

I expect that the cue loader would:

  • ignore symlinks and other irregular files

Please can you explain why this is your expectation with respect to symlinks?

@myitcv
Copy link
Member

myitcv commented Apr 29, 2022

Crucial edit in my previous reply :)

@myitcv myitcv added the zGarden label Jun 13, 2023
@mvdan mvdan removed the zGarden label Feb 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cue: ignore bad symlinks and unusable file types kcburge/cue
ignore irregular files kcburge/cue
5 participants
@rogpeppe @kcburge @verdverm @myitcv @mvdan