First off, I agree that this is unintentional and thereby a bug.

I do wonder if an alternative solution could be to update the filtering function to not filter out directories that contain files that would ultimately be included. It looks like we inherit some gitignore functionality from a library so it may not be as trivial as I make it sound but it may be worth exploring. The main benefit being that there's no post-processing necessary.

I'd like to work on this if nobody else isn't.
I just needed to clear up a few things here -
I understand the issue that happens with include part of the bug, but the exclude part is still unclear.
From my understanding if the same project.toml definition is used with exclude = ["testdir/testfile"] then the testfile will not be copied but contrary to the issue pointed out above the directory testdir is owned by cnb instead of root.

===> BUILDING
.:
total 16K
drwsrwsrwt 3 cnb  cnb  4.0K Jan  1  1970 .
drwxr-xr-x 1 root root 4.0K May  8 12:54 ..
-rw-r--r-- 1 cnb  cnb   299 May  8 12:54 project.toml
drwxr-xr-x 2 cnb  cnb  4.0K May  8 12:53 testdir

./testdir:
total 8.0K
drwxr-xr-x 2 cnb cnb 4.0K May  8 12:53 .
drwsrwsrwt 3 cnb cnb 4.0K Jan  1  1970 ..

I wanted to confirm if this is how the bug is triggered for exclude, or let me know if I am missing something here.

For the include part of the bug I agree that we can update the filtering function to include all the parent directories of files that are specified inside include

# If include contains
include = ["testdir/abc/testfile", "dir2/file.txt"]

# Get parent dirs
parent_dirs = ["testdir", "abc", "dir2"]

# Add their respective gitignore entries to `fileFilter`

I can start working on the include part if the approach seems fine, moreover exclude can be worked on after clarifications.

@imnitishng It makes sense that your example of excludes works as expected because you are excluding the file and not the directory so therefore the directory wouldn't match the "ignore". @BenjaminSchubert if you could provide some examples on how this could be recreated via exclude that would be appreciated otherwise will move forward with fixing include.

For include, the pseudo strategy feels a little incomplete. That may just be my misunderstanding on how it's written. I'm concerned about partial includes for parent directories (ie. abc in the example above). If we only put the directory name of abc then any other directory with that name (elsewhere) may be included.

So in pseudo terms it should really be more like this:

# If include contains
include = ["testdir/abc/testfile", "dir2/file.txt"]

# Get parent dirs
parent_dirs = [
  "./testdir/", 
  "./testdir/abc/", 
  "./dir2/",
]

Let me know if you agree or if I'm missing something.

I'm concerned about partial includes for parent directories (ie. abc in the example above). If we only put the directory name of abc then any other directory with that name (elsewhere) may be included.

Yea, the example I provided was quite vague, but I had the same idea in my mind. I will start working on this and create a draft PR for include and extend that once we get exclude clear.
Will make sure tests are robust enough to deal with any edge cases that may happen.

Hey @jromero, sure, something like:

[build]
# This triggers the bug
exclude = ["testdir", "!testdir/testfile"]

[[build.buildpacks]]
id = "testing/workspace-info"

  [build.buildpacks.script]
  api = "0.7"
  # The build will always fail, but we care about the output here
  inline = "ls -lahR && exit 1"

should work though I can't test it right now

@BenjaminSchubert ah, that's an interesting edge case. I feel conflicted by it. Is it considered a misconfiguration given that the nested file obviously needs the parent directories?

Some options that come to mind:

1. Do nothing - this feels like an edge case that can be easily explained when configuration is looked at.
2. "Fix" for the user - we could add some logic to remove any parent directories from excludes if children files/directory are being included. This feels like it could be very error prone if their intentions were different.
3. Warning only - if we detect that a parent directory is being excluded while a children file/directory is being included.

I'm currently leaning towards 1) because the added complexity doesn't feel justified.

If the intent of the user is to exclude everything in the testdir except for a single file then it should probably be: [testdir/*, !testdir/file]

Reference: https://stackoverflow.com/a/29932318

@jromero sorry for the delay in responding.

@BenjaminSchubert ah, that's an interesting edge case. I feel conflicted by it. Is it considered a misconfiguration given that the nested file obviously needs the parent directories?

git does work correctly with such patterns, and the spec mentions that it follow the gitignore pattern here. You can for example, with git do:

mysubdir/
!mysubdir/myfile

and that does work (well, technically git doesn't really care about directories at all).

So if we wanted to consider it a misconfiguration, we should document and catch this case as the build goes, with a nice error message instead. I don't think "just letting it go through" is a nice user experience.

I would more lean towards 2, or a potential number 4, which would be:

4. Error if such a pattern is found. This is not going to give the expected results anyways

And I was wrong.

I've played more with git, and it does look like

mysubdir/
!mysubdir/file

does ignore mysubdir/file, so number 2 would actually deviate from the gitignore way of doing things.

Which means, it's handled however incorrectly in pack... as it does add the file.

At that point, I think an error with message might be the best bet