Add new filtering fs: FilePredicateFs

[satotake]

While golang's built-in regexp package has the advantage, constant time guarantees,
it (RE2) has some limitations to express complex rules compared to other languages (PCRE2).
(For example, it does not have backtracking.)
Here, I would like to introduce a new filtering backend, PredicateFs, to make afero's filtering more flexible and composable.

PredicateFs is a filtered view on predicates which takes file path as an argument.
Any file will be treated as non-existing when the predicate returns false.
Unlike RegexpFs, the fs targets not file name but file path.
Like, RegexpFs, files will not be created when the predicate returns false and directories are always not filtered.

[CLAassistant]

All committers have signed the CLA.

[jxsl13]

This one is pretty nice as it may filter according to any kind of path related rule, not just a regular expression.

You removed a #from the readme of HttpFs

Imo it shouldn't handle directories in any special way.
Might as well provide access to the internal Fs in the predicate function in order to e.g. check if the file is a directory or do other checks with the file path. This might need some synchronozation mechanism, I'd guess or one assumes that the Fs is goroutine safe.
I wonder what the thoughts behind that reasoning were?
Same as RegexpFs?

I could think of a use case where I want my Fs to be only able to interact within specific directories somewhat like BasePathFs but with multiple directories e.g. provided in a regular expression or in a list.

[satotake]

You removed a #from the readme of HttpFs

This is not related with this request directly but I think HttpFs is not filtering fs.

I wonder what the thoughts behind that reasoning were?
Same as RegexpFs?

Good point.
Though I am not sure of the rationales of RegexpFs, probably they are the same.

Consider the situation where we

• want to include only files with .go extension
• use fs.WalkDir defined walk function, which calls Open internally

(yes, this will be minor)

If we apply the function with both files and dirs, strings.HasSuffix(s, ".go") will be wrong for files in sub dir.
dir/example.go is skipped surprisingly(?) because strings.HasSuffix("dir", ".go") is false.
This is easy to go wrong in my opinion.

Instead of it, as you describe, it will be clear to combine PredicateFs with BasepathFs etc.
Or, we can handle directories as I wrote the test though it is a little verbose (.hidden).

[jxsl13]

In case that you provide the underlying Fs as parameter to the predicate function, you might be able to check whether the file path is actually a file or a directory with a Stat call and then go for the file's suffix.

^
I think that this would make things way more complicated, for sure and potentially more error prone.

Might as well rename PredicateFs to FilePredicateFs in order to make it clearer that this abstraction is only targeting files.

[satotake]

@jxsl13 thanks for reviewing.
Replaced PredicateFs with FilePredicateFs

[bep]

Sorry for being very late to the game, this looks very useful. I have read the discussion above, but I suspect that we're missing out on some potential when we apply this to files (and not dirs) only. If I walk a directory I certainly want to be able to skip a dir before reading it.

How about making the predicate:

func(isDir bool, path string) bool

[satotake]

@bep thank you.
I referred to RegexpFs and it skips all directories internally.
So does this fs.

[bep]

@satotake I'm not sure what you mean by "skip directories"; to me it looks like every directory is included.

I would have found it much more useful if the above was replaced with:

if f.pred(i.IsDir(), filepath.Join(f.f.Name(), i.Name())) {

[jxsl13]

One might as well just pass the file info object instead of a bool which can then be manually checked whether the path is a directory or a file or a symlink, etc.

[bep]

That would break if you want to use this with fs.ReadDirFile (see my recent commit to the main branch).

[satotake]

@bep I got your point.
isDir will be nice to ignore some dirs without scanning their children, for instance.
Fixed.