Added Include/Exclude filtering capability to Unzip Task (#5169) #6018

IvanLieckens · 2021-01-11T14:36:55Z

Context

Changes Made

Unzip Task now has "Include" and "Exclude" optional properties to pass a pattern to filter archive entries to be unzipped.

Testing

Added following tests:

CanUnzip_WithIncludeFilter
CanUnzip_WithExcludeFilter
CanUnzip_WithIncludeAndExcludeFilter

These 3 test the ability to include/exclude files from the archive unzip.

Notes

Unable to translate the resources to all languages. Can someone provide guidance/translations?

rainersigwald · 2021-01-11T17:40:21Z

Don't worry about the failing PR builds--I believe #6019 will fix that.

Unable to translate the resources to all languages. Can someone provide guidance/translations?

Thanks, but no need to worry! A Microsoft-funded team will do the translations. Details in the loc doc if you're interested.

rainersigwald · 2021-01-11T22:04:38Z

/azp run

azure-pipelines · 2021-01-11T22:04:47Z

Azure Pipelines successfully started running 1 pipeline(s).

Forgind

LGTM!

Forgind · 2021-01-13T19:48:14Z

src/Tasks/Resources/Strings.resx

@@ -2792,6 +2792,9 @@
  <data name="Unzip.DidNotUnzipBecauseOfFileMatch">
    <value>Did not unzip from file "{0}" to file "{1}" because the "{2}" parameter was set to "{3}" in the project and the files' sizes and timestamps match.</value>
  </data>
+  <data name="Unzip.DidNotUnzipBecauseOfFilter">
+    <value>Did not unzip file "{0}" because it didn't match the include or matched the exclude filter.</value>


nit:

Suggested change

<value>Did not unzip file "{0}" because it didn't match the include or matched the exclude filter.</value>

<value>Did not unzip file "{0}" because it didn't match the include or because it matched the exclude filter.</value>

No problem, I have applied it in the new commits

Forgind · 2021-01-13T19:49:57Z

src/Tasks.UnitTests/Unzip_Tests.cs

+            using (TestEnvironment testEnvironment = TestEnvironment.Create())
+            {
+                TransientTestFolder source = testEnvironment.CreateFolder(createFolder: true);
+                TransientTestFolder destination = testEnvironment.CreateFolder(createFolder: false);


Would you mind modifying this test to use wildcards such that two are included, of which one is also excluded; a third is also excluded; and a fourth isn't excluded or included?

Not at all, I hope this new version is more what you were looking for?

benvillalobos · 2021-01-15T16:51:23Z

ref/Microsoft.Build.Tasks.Core/net/Microsoft.Build.Tasks.Core.cs

@@ -1217,6 +1217,8 @@ public sealed partial class Unzip : Microsoft.Build.Tasks.TaskExtension, Microso
        public Unzip() { }
        [Microsoft.Build.Framework.RequiredAttribute]
        public Microsoft.Build.Framework.ITaskItem DestinationFolder { get { throw null; } set { } }
+        public string Exclude { get { throw null; } set { } }
+        public string Include { get { throw null; } set { } }


If we stick with regular expressions for this, we should change the names for Exclude and Include as they are "default" msbuild names. ie. items are added via Include and the patterns differ between them and regular expressions. A suggested name during PR review was IncludePattern & ExcludePattern

This makes sense, the functionality differs indeed. I have pushed the change.

@benvillalobos, I thought we'd agreed globs were more MSBuild-y than regex? Confused by your comment.

We did, I posted this while we were talking about it, hence the "If we stick with regular expressions." The current implementation is still regex, is this how we process includes and excludes normally?

No; we use globs.

@benvillalobos ok, thank you for the clearing up. Is there a good example in MSBuild of how you'd like this to function I can use as a lead for the implementation? Just want to make sure that it feels native without any quirks.

Check out Expander.cs, ExpandIntoItemsLeaveEscaped may have what you're looking for.

ItemSpec.cs is also relevant here.

Thank you @benvillalobos , I studied Expander and ItemSpec but they were outside of reach for the Tasks assembly and I did not want to introduce dependencies so I leveraged FileMatcher to handle the normalization and verification of the globs and paths. It does not support property references due to this. I hope this matches with your expected behavior.

MSBuildGlob would have been great here, but unfortunately it's not visible by tasks. :(

…per Review Suggestion

benvillalobos

The PR overall looks good, pending the switch from regex to globs.

benvillalobos · 2021-01-20T23:45:15Z

ref/Microsoft.Build.Tasks.Core/net/Microsoft.Build.Tasks.Core.cs

@@ -1217,6 +1217,8 @@ public sealed partial class Unzip : Microsoft.Build.Tasks.TaskExtension, Microso
        public Unzip() { }
        [Microsoft.Build.Framework.RequiredAttribute]
        public Microsoft.Build.Framework.ITaskItem DestinationFolder { get { throw null; } set { } }
+        public string Exclude { get { throw null; } set { } }
+        public string Include { get { throw null; } set { } }


So we're in agreement 🙂

@IvanLieckens the to-do list here would be:

Rename ExcludePattern and IncludePattern back to Exclude and Include (sorry)

We wanted ExcludePattern if we were sticking with RegEx, but it turns out we don't want to use regex here

Change the implementation to expect * parse a glob pattern

…o regular glob matching instead of RegEx

benvillalobos · 2021-01-27T21:15:07Z

src/Shared/FileMatcher.cs

@@ -31,6 +31,8 @@ internal class FileMatcher
        private static readonly char[] s_wildcardCharacters = { '*', '?' };
        private static readonly char[] s_wildcardAndSemicolonCharacters = { '*', '?', ';' };

+        private static readonly string[] s_propertyReferences = { "$(", "@(" };


Suggested change

private static readonly string[] s_propertyReferences = { "$(", "@(" };

private static readonly string[] s_propertyAndItemReferences = { "$(", "@(" };

Items are referred to with an @ and properties with $.

Thanks, I just copied from the old name of the method. I adjusted the naming now to these.

benvillalobos · 2021-01-27T21:17:33Z

src/Shared/FileMatcher.cs

+        /// <summary>
+        /// Determines whether the given path has any property references.
+        /// </summary>
+        internal static bool HasPropertyReferences(string filespec)


Suggested change

internal static bool HasPropertyReferences(string filespec)

internal static bool HasPropertyOrItemReferences(string filespec)

Another option would be to create a HasPropertyReferences that checks for $( and a separate HasItemReferences that checks for @(. Though I don't feel too strongly about that extra suggestion since filematcher already has something like s_wildcardAndSemicolonCharacters and HasWildcardsOrSemicolon.

Yeah, I split up the existing method that was already there to allow more granular calling and identification. But I didn't want to introduce too many new methods if they weren't needed.

benvillalobos · 2021-01-27T21:19:11Z

src/Shared/FileMatcher.cs

+        /// </summary>
+        internal static bool HasWildcardsOrSemicolon(string filespec)
+        {
+            return -1 != filespec.LastIndexOfAny(s_wildcardAndSemicolonCharacters);


Is there any significant perf difference between using Aggregate and -1 != filespec.LastIndexOfAny(s_wildcardAndSemicolonCharacters)?

Were you thinking Aggregate with a function like Aggregate(false, (acc, ch) => acc || s_wildcardAndSemicolonCharacters.Contains(ch))? That would almost certainly be slower than this.

I was thinking the difference between s_propertyReferences.Aggregate(false, (current, propertyReference) => current | filespec.Contains(propertyReference)); and -1 != filespec.LastIndexOfAny(s_propertyReferences);

If the former is more efficient, we can change HasWildcardsOrSemicolon to do the same.

I believe LastIndexOf is going to be faster but I had to use the aggregate for the other because the s_wildcardAndSemicolonCharacters are char[] and the s_propertyReferences are string[]. Would need to setup a microbenchmark to validate.

Ah! I didn't realize that we couldn't replicate what was already there. This looks fine to me 👍

Forgind improved it further now to use Any() :)

src/Shared/FileMatcher.cs

Forgind · 2021-01-28T16:32:12Z

src/Tasks/Unzip.cs

+                patterns = pattern.Contains(';')
+                               ? pattern.Split(new[] { ';' }, StringSplitOptions.RemoveEmptyEntries).Select(FileMatcher.Normalize).ToArray()
+                               : new[] { pattern };
+                if (patterns.Any(p => p.IndexOfAny(Path.GetInvalidPathChars()) != -1))


nit:
Move this before the split?

This is very strange...the docs for Path.GetInvalidPathChars explicitly says that "on Windows-based desktop platforms, invalid path characters might include...less than (<), greater than (>), pipe (|),..." yet trying that out on my Windows-based desktop platform, it didn't. I submitted an issue about it: dotnet/dotnet-api-docs#5292

I'd use FileUtilities.InvalidPathChars and modify the tests to target | or a character 1-31 and make it the same across all platforms.

That's very interesting, I changed to using the FileUtilities.InvalidPathChars and added a pipe in the name in the test whilst removing the platform specific flag.

src/Tasks/Unzip.cs

Forgind · 2021-01-28T16:44:48Z

src/Tasks/Unzip.cs

@@ -212,5 +275,41 @@ private bool ShouldSkipEntry(ZipArchiveEntry zipArchiveEntry, FileInfo fileInfo)
                   && zipArchiveEntry.LastWriteTime == fileInfo.LastWriteTimeUtc
                   && zipArchiveEntry.Length == fileInfo.Length;
        }
+
+        private bool ParseIncludeExclude()


Why does this return anything? Looking at ParsePattern below, it either throws an error or returns true. That means that this either throws an error or returns true. I was momentarily confused when I thought it was skipping everything if there were no include/exclude present, so I'd just remove that bit.

Ok, I removed the return and swapped to using Log.HasLoggedErrors to jump out preventing further execution when the Task is misconfigured.

Co-authored-by: Forgind <Forgind@users.noreply.github.com>

…n Unzip unit tests

Co-authored-by: Forgind <Forgind@users.noreply.github.com>

…of Unzip Task

Forgind

LGTM! Thanks for bearing with us on this!

Forgind · 2021-01-29T16:57:58Z

src/Shared/FileMatcher.cs

@@ -204,7 +204,7 @@ internal static bool HasWildcardsSemicolonItemOrPropertyReferences(string filesp
        /// </summary>
        internal static bool HasPropertyOrItemReferences(string filespec)
        {
-            return s_propertyAndItemReferences.Any(ref=> filespec.Contains(ref));
+            return s_propertyAndItemReferences.Any(filespec.Contains);


src/Tasks/Unzip.cs

Co-authored-by: Mihai Codoban <micodoba@microsoft.com>

Forgind · 2021-02-06T21:03:03Z

Thanks @IvanLieckens!

sc-ivanlieckens added 2 commits January 11, 2021 15:21

Added Include/Exclude RegEx filtering capability to Unzip Task

9987940

Added Exclude Filter Unit Test for Unzip Task

08cb0d1

Forgind approved these changes Jan 13, 2021

View reviewed changes

sc-ivanlieckens added 2 commits January 14, 2021 19:09

Adjusted Unit Test according to feedback

86c25c2

Applied resource string suggestion

fbc0a4f

benvillalobos reviewed Jan 15, 2021

View reviewed changes

Adjusted Include and Exclude to IncludePattern and ExcludePattern as …

e71e37f

…per Review Suggestion

benvillalobos reviewed Jan 20, 2021

View reviewed changes

sc-ivanlieckens added 2 commits January 26, 2021 19:31

Adjusted property names to "Include" and "Exclude" and altered them t…

152aadc

…o regular glob matching instead of RegEx

Unzip task unit tests for pattern parsing

941d7eb

benvillalobos reviewed Jan 27, 2021

View reviewed changes

sc-ivanlieckens added 3 commits January 28, 2021 08:53

Name adjustment to better match actual verification

658c23a

Attempt to fail valid character path tests on MacOS and Linux

8164bfd

Made InvalidPath Unzip unit tests platform specific

1f1aa6d

Forgind reviewed Jan 28, 2021

View reviewed changes

IvanLieckens and others added 7 commits January 29, 2021 09:31

Improved property or item reference detection

5e489e2

Co-authored-by: Forgind <Forgind@users.noreply.github.com>

Invalid path detection before splitting improvement

9fedb34

Added | to invalid path testing

62d56a9

Removed platform specific attribute on Invalid Path Character tests i…

76a94f1

…n Unzip unit tests

Perf improvement detecting include pattern match

e9e0f5f

Co-authored-by: Forgind <Forgind@users.noreply.github.com>

Perf improvement detecting exclude pattern match

8608320

Co-authored-by: Forgind <Forgind@users.noreply.github.com>

Small FileMatcher correction, Removed return on parsing for patterns …

e4d0286

…of Unzip Task

IvanLieckens changed the title ~~Added Include/Exclude RegEx filtering capability to Unzip Task (#5169)~~ Added Include/Exclude filtering capability to Unzip Task (#5169) Jan 29, 2021

Forgind approved these changes Jan 29, 2021

View reviewed changes

cdmihai approved these changes Jan 29, 2021

View reviewed changes

cdmihai reviewed Jan 29, 2021

View reviewed changes

src/Tasks/Unzip.cs Outdated Show resolved Hide resolved

cdmihai reviewed Jan 29, 2021

View reviewed changes

src/Tasks/Unzip.cs Outdated Show resolved Hide resolved

IvanLieckens and others added 2 commits January 30, 2021 00:17

Updated Include description

164e8e7

Co-authored-by: Mihai Codoban <micodoba@microsoft.com>

Updated Exclude description

081c179

Co-authored-by: Mihai Codoban <micodoba@microsoft.com>

Forgind added the merge-when-branch-open PRs that are approved, except that there is a problem that means we are not merging stuff right now. label Jan 29, 2021

Forgind merged commit 70f6767 into dotnet:master Feb 6, 2021

IvanLieckens deleted the feature/UnzipFiltering branch February 7, 2021 10:19

	<value>Did not unzip file "{0}" because it didn't match the include or matched the exclude filter.</value>
	<value>Did not unzip file "{0}" because it didn't match the include or because it matched the exclude filter.</value>

	private static readonly string[] s_propertyReferences = { "$(", "@(" };
	private static readonly string[] s_propertyAndItemReferences = { "$(", "@(" };

	internal static bool HasPropertyReferences(string filespec)
	internal static bool HasPropertyOrItemReferences(string filespec)

Added Include/Exclude filtering capability to Unzip Task (#5169) #6018

Added Include/Exclude filtering capability to Unzip Task (#5169) #6018

Conversation

IvanLieckens commented Jan 11, 2021 • edited

Context

Changes Made

Testing

Notes

rainersigwald commented Jan 11, 2021

rainersigwald commented Jan 11, 2021

azure-pipelines bot commented Jan 11, 2021

Forgind left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Forgind Jan 20, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benvillalobos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Forgind left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Forgind commented Feb 6, 2021

IvanLieckens commented Jan 11, 2021 •

edited

Forgind Jan 20, 2021 •

edited