Add support for aliased cultures #6003

0xced · 2021-01-04T15:13:46Z

rainersigwald

Thanks! This change looks good to me (except for the minor nitpick comments I made) but I don't know too much about this aspect of localization. @wli3, @tarekgh, can I get y'all to take a look?

rainersigwald · 2021-01-04T16:56:24Z

src/Tasks/CultureInfoCache.cs

+            // See https://docs.microsoft.com/en-us/dotnet/api/System.Globalization.CultureInfo.LCID#remarks
+            const int LOCALE_CUSTOM_UNSPECIFIED = 0x1000;
+            if (culture.LCID == LOCALE_CUSTOM_UNSPECIFIED)


Can you please expand on this a bit? It's not immediately obvious to me from the link why non-system locales should be fully rejected here.

Actually, this could be further discussed. What do we want? Accept any valid culture or accept known (by the operating system) cultures? If we accept any valid culture then we can get rid of this check altogether.

src/Tasks/CultureInfoCache.cs

rainersigwald · 2021-01-04T17:06:15Z

src/Tasks.UnitTests/AssignCulture_Tests.cs

+
+            Assert.Single(t.AssignedFiles);
+            Assert.Single(t.CultureNeutralAssignedFiles);
+            Assert.Equal(String.Empty, t.AssignedFiles[0].GetMetadata("Culture"));


This is the critical difference between this test and the above, right? Can you call it out with a comment?

Do you mean that the Culture metadata is empty?

In general I just mean "Please call out the parts of this test that are different". Here I guess it's not just that Culture is empty but also that the culture-neutral part doesn't get assigned.

tarekgh · 2021-01-04T17:32:08Z

I believe we should get rid of the whole cache and just use CultureInfo.GetCultureInfo API. are we not using this API for any perf reason?

Note that on Windows 10 (on on Linux using ICU), all cultures with the well formed name is considered a valid culture even if the system doesn't have a data for. why we are trying to create a cache at all? CultureInfo.GetCultureInfo already maintaining internal cache for all previously created cultures.

rainersigwald · 2021-01-04T17:48:14Z

I believe we should get rid of the whole cache and just use CultureInfo.GetCultureInfo API. are we not using this API for any perf reason?

The .NET Core version of this was introduced in #213, but that was a patch over CultureInfo.GetCultures not existing in .NET Core 1.0. The Framework code that relies on the enumeration and caching dates back to 2003-11-09 18:07:58 in a commit that has an entirely useless message but also has this change:

-                try
-                {
-                    // Construct a CultureInfo just to see if its possible.
-                    new CultureInfo(cultureName);
-                    validCulture = true;
-                }
-                catch (ArgumentException)
-                {
-                    // Just eat the exception, but now we know the culture isn't valid. 
-                }
+                validCulture = CultureStringUtilities.IsValidCultureString(cultureName);

So maybe there was a perf issue 18 years ago? I certainly like the simplicity of your proposal.

tarekgh · 2021-01-04T17:56:19Z

@rainersigwald we should give it a try. I think the perf hit would be in invalid cases only which I think is not common. Also, on Windows 10, most of the time will not encounter any invalid cases too.

wli3 · 2021-01-04T18:08:36Z

I am not familiar with this issue. I will pass it to @tarekgh :)

rainersigwald · 2021-01-04T18:17:24Z

@rainersigwald Rainer Sigwald FTE we should give it a try. I think the perf hit would be in invalid cases only which I think is not common. Also, on Windows 10, most of the time will not encounter any invalid cases too.

Ok. @0xced would you be willing to make that change?

0xced · 2021-01-04T19:45:05Z

Sure. I force-pushed and simplified to the maximum! (+77 / -901) The IsValidCultureString method is now just a few lines, anything "smart" has been removed.

internal static bool IsValidCultureString(string name)
{
    try
    {
        _ = CultureInfo.GetCultureInfo(name);
        return true;
    }
    catch (CultureNotFoundException)
    {
        return false;
    }
}

Fixes dotnet#3897

Co-authored-by: Rainer Sigwald <raines@microsoft.com>

…rity Co-authored-by: Rainer Sigwald <raines@microsoft.com>

Remove anything "smart" and only rely on `CultureInfo.GetCultureInfo` not throwing `CultureNotFoundException` to deem a culture name valid. Since CultureInfo.GetCultureInfo() returns a cached CultureInfo there should be no performance issue.

tarekgh · 2021-01-04T20:25:37Z

@0xced thanks a lot for your help here. the changes looks good. could you please try to clean the usage of FEATURE_CULTUREINFO_GETCULTURES from the cs project and also if there is dead code (maybe something like AssemblyUtilities.CultureInfoHasGetCultures()).

@rainersigwald do we have any perf test can be used to ensure we are not regressing the perf with that?

This was needed in the .NET Core 1.0 era but CultureInfo.GetCultures and CultureInfo.GetCultureInfo are now available in .NET Core (since version 2.0).

rainersigwald · 2021-01-04T20:41:44Z

@rainersigwald do we have any perf test can be used to ensure we are not regressing the perf with that?

cc @ladipro and @Forgind but not really. We have VS RPS tests but they exercise very small projects and I don't know if they touch this codepath at all.

0xced · 2021-01-04T20:47:42Z

I did some cleanup on FEATURE_CULTUREINFO_GETCULTURES and FEATURE_CULTUREINFO_GETCULTUREINFO in 55f8fee.

When doing this cleanup I noticed this comment:

// Assembly.Location is only available in .netstandard1.5, but MSBuild needs to target 1.3.
// use reflection to access the property

Is it still true that MSBuild needs to target .NET Standard 1.3?

rainersigwald · 2021-01-04T20:56:16Z

Is it still true that MSBuild needs to target .NET Standard 1.3?

Nope! Please feel free to remove that :)

Forgind · 2021-01-04T21:07:49Z

It looks like at some point they decided to throw an error when it hadn't been initialized properly? Not sure why:

msbuild/src/MSBuild.UnitTests/CommandLineSwitches_Tests.cs

Lines 934 to 982 in c70d520

    
                   [Fact] 
        
                   public void InvalidToolsVersionErrors() 
        
                   { 
        
                       Assert.Throws<InitializationException>(() => 
        
                       { 
        
                           string filename = null; 
        
                           try 
        
                           { 
        
                               filename = FileUtilities.GetTemporaryFile(); 
        
                               ProjectRootElement project = ProjectRootElement.Create(); 
        
                               project.Save(filename); 
        
                               MSBuildApp.BuildProject( 
        
                                                   filename, 
        
                                                   null, 
        
                                                   "ScoobyDoo", 
        
                                                   new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase), 
        
                                                   new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase), 
        
                                                   new ILogger[] { }, 
        
                                                   LoggerVerbosity.Normal, 
        
                                                   new DistributedLoggerRecord[] { }, 
        
           #if FEATURE_XML_SCHEMA_VALIDATION 
        
                                                   false, 
        
                                                   null, 
        
           #endif 
        
                                                   1, 
        
                                                   true, 
        
                                                   new StringWriter(), 
        
                                                   new StringWriter(), 
        
                                                   false, 
        
                                                   warningsAsErrors: null, 
        
                                                   warningsAsMessages: null, 
        
                                                   enableRestore: false, 
        
                                                   profilerLogger: null, 
        
                                                   enableProfiler: false, 
        
                                                   interactive: false, 
        
                                                   isolateProjects: false, 
        
                                                   graphBuild: false, 
        
                                                   lowPriority: false, 
        
                                                   inputResultsCaches: null, 
        
                                                   outputResultsCache: null 
        
                                   ); 
        
                           } 
        
                           finally 
        
                           { 
        
                               if (File.Exists(filename)) File.Delete(filename); 
        
                           } 
        
                       } 
        
                      ); 
        
                   }

It goes back to the original commit:
82177a5#diff-0ef03e858b5dafafd31e725cf2f7be2c22aaa57214afc6bee56d639652b3a9d5R1066-R1082

benvillalobos

So the idea behind the change is to stop dynamically generating valid cultures and instead use the CultureInfo API (which results in supporting aliased cultures)? This change generally LGTM.

benvillalobos · 2021-01-05T01:03:38Z

src/Tasks/CultureInfoCache.cs

-            return ValidCultureNames.Contains(name);
+            try
+            {
+                _ = CultureInfo.GetCultureInfo(name);


We're going from a hashset lookup to try-catching a function call. This will run for every resource in a project, what's the perf impact here?

Looks like it does a TryGetValue on a dictionary behind the scenes. https://source.dot.net/#System.Private.CoreLib/CultureInfo.cs,fec605b3b773ab26

I didn't notice the discussion before, it looks like we're okay with this change so long as RPS doesn't catch this.

I also didn't realize that the previous valid cultures are calculated on each run. This seems fine to me.

Try itself is fast as long as it doesn't hit the catch (StackOverflow), and I imagine the cache is small enough that the dictionary lookup will be faster than looking through a list of valid cultures. My biggest worry would be if calling into the runtime causes us to load an assembly we otherwise didn't repeatedly. Not sure about that.

My biggest worry would be if calling into the runtime causes us to load an assembly we otherwise didn't repeatedly. Not sure about that.

The Globalization code residing inside the core library. so this change wouldn't make any difference. Or are you referring to something else?

0xced · 2021-01-05T09:24:23Z

I'm a bit lost as to why the InvalidToolsVersionErrors test is now failing and how it relates to the change introduced in this pull request. I noticed that the tests fail only when run with eng/cibuild_bootstrapped_msbuild.sh (which is using stage1/bin/bootstrap/netcoreapp2.1/MSBuild/MSBuild.dll) but I'm not sure what this means. All tests are passing when I run ./build.sh --test --ci (which is using .dotnet/sdk/3.1.100/MSBuild.dll).

Can someone with more MSBuild experience help me to diagnose this issue?

benvillalobos · 2021-01-06T00:37:20Z

I wonder if the netcoreapp2.1 MSBuild.dll referenced is using an older version of whatever assembly CultureInfo.GetCultureInfo comes from, and that older version doesn't include aliased cultures? I'll dig into this a bit and report any findings.

edit: I'm constantly running into an issue with eng\cibuild_bootstrapped_msbuild.sh not running the second build because of a permission issue so I can't get a local repro.

benvillalobos

Shouldly errors read better, we should switch to those while we're modifying these functions.

benvillalobos · 2021-01-06T01:37:05Z

src/Build.UnitTests/Evaluation/Expander_Tests.cs

            string result = expander.ExpandIntoStringLeaveEscaped(@"$([System.Globalization.CultureInfo]::GetCultureInfo(`en-US`).ToString())", ExpanderOptions.ExpandProperties, MockElementLocation.Instance);
-#else
-            string result = expander.ExpandIntoStringLeaveEscaped(@"$([System.Globalization.CultureInfo]::new(`en-US`).ToString())", ExpanderOptions.ExpandProperties, MockElementLocation.Instance);
-#endif

            Assert.Equal(new CultureInfo("en-US").ToString(), result);


While we're here

Suggested change

Assert.Equal(new CultureInfo("en-US").ToString(), result);

result.ShouldBe(new CultureInfo("en-US").ToString());

benvillalobos · 2021-01-06T01:38:29Z

src/Tasks.UnitTests/AssignCulture_Tests.cs

@@ -216,6 +256,30 @@ public void PseudoLocalization(string culture)
            Assert.Equal($"MyResource.{culture}.resx", t.AssignedFiles[0].ItemSpec);


This applies to the other asserts as well, we should shouldly-ify functions we touch on.

Suggested change

Assert.Equal($"MyResource.{culture}.resx", t.AssignedFiles[0].ItemSpec);

t.AssignedFiles[0].ItemSpec.ShouldBe($"MyResource.{culture}.resx");

benvillalobos · 2021-01-06T01:38:42Z

src/Tasks.UnitTests/AssignCulture_Tests.cs

+            t.Files = new ITaskItem[] { i };
+            t.Execute();
+
+            Assert.Single(t.AssignedFiles);


Assert -> Shouldly function calls.

Accidentally approved

0xced · 2021-01-06T13:22:26Z

Here's why the tests were failing: MSBuild itself uses Strings.shared.resx as resource name. When accepting shared as a valid culture, Microsoft.Build.Strings.shared.resources would not end up in the embedded resources of artifacts/bin/Microsoft.Build.Engine.UnitTests/Debug/netcoreapp2.1/Microsoft.Build.dll leading to the MissingManifestResourceException that was the cause of the failing tests.

MSBuild itself is a good example as to why we should be conservative and accept only aliased cultures in order not to break projects using Name.something.resx where something is not a locale.

I have improved this in 4bdca75 to only deem aliased cultures as valid, not all cultures.

0xced · 2021-01-06T13:30:51Z

Thinking more about it, maybe we hit another bug because if shared was assigned as a culture then it should have ended as Microsoft.Build.Strings.shared.resources in the embedded resources of artifacts/bin/Microsoft.Build.Engine.UnitTests/Debug/netcoreapp2.1/Microsoft.Build.dll. 🤔

On .NET Framework, ThreeLetterISOLanguageName has two letters for unknown cultures such as "xx".

…ework

0xced · 2021-01-07T21:51:44Z

So, I think I finally found a way to identify predefined cultures, under both ICU and NLS.

To deem a culture valid it must either:

Have a LCID != 4096 (LOCALE_CUSTOM_UNSPECIFIED)
or
Have a ThreeLetterISOLanguageName and have a NativeName which does not start with "Unknown Language"

A better solution would be to use CultureInfo.GetCultureInfo(name, predefinedOnly: true) but unfortunately it is only available since .NET 5.

I also tweaked the test so that they pass on all supported platforms. Supported cultures vary a lot across different platforms!

tarekgh · 2021-01-07T21:59:23Z

Have a ThreeLetterISOLanguageName and have a NativeName which does not start with "Unknown Language"

Please don't use this. Windows is not promising this will work. That is why we have exposed the other GetCultureInfo overload.

Have a LCID != 4096 (LOCALE_CUSTOM_UNSPECIFIED)

If you use this, please check LOCALE_CUSTOM_DEFAULT too.

LOCALE_CUSTOM_DEFAULT       = 0x0c00;

Also to clarify, checking LOCALE_CUSTOM_UNSPECIFIED would exclude some legitimate Windows cultures. I didn't follow closely here but would be nice if someone clarify why we care filtering out these cultures?

0xced · 2021-01-20T19:58:25Z

Please don't use this. Windows is not promising this will work. That is why we have exposed the other GetCultureInfo overload.

Hmmm, but CultureInfo.GetCultureInfo(name, predefinedOnly: true) is only available since .NET 5 and I don't think MSBuild is ready to switch to .NET 5, right? Is there another way without using a .NET 5-only API?

Also to clarify, checking LOCALE_CUSTOM_UNSPECIFIED would exclude some legitimate Windows cultures.

Do you have an example of such a culture so that I can add it to the tests and make sure it's considered a valid culture?

I didn't follow closely here but would be nice if someone clarify why we care filtering out these cultures?

Just to be conservative and to not accidentally break too much existing projects. MSBuild itself breaks when using shared as a culture name, see my previous comments: #6003 (comment)

tarekgh · 2021-01-20T20:24:03Z

Hmmm, but CultureInfo.GetCultureInfo(name, predefinedOnly: true) is only available since .NET 5 and I don't think MSBuild is ready to switch to .NET 5, right? Is there another way without using a .NET 5-only API?

I assume we are talking when running on Windows. If so, you may call the OS directly for that. something like the call https://source.dot.net/#System.Private.CoreLib/CultureInfo.Nls.cs,14
If we need that on Linux, that will be more complicated to do as we depend on ICU to get the info.

Do you have an example of such a culture so that I can add it to the tests and make sure it's considered a valid culture?

please run the following code and it will give you such list.

            foreach (CultureInfo ci in CultureInfo.GetCultures(CultureTypes.AllCultures))
            {
                if (ci.LCID == 0x1000)
                {
                    Console.WriteLine(ci.Name);
                }
            }

Just to be conservative and to not accidentally break too much existing projects. MSBuild itself breaks when using shared as a culture name, see my previous comments: #6003 (comment)

Is there any other way to include the shared resources without considering it as a culture? I mean can the resource file be renamed to something else (like String_shared.resx for instance)? or we don't have control over that?

Forgind · 2021-03-31T17:05:42Z

Any recent updates on this?

Forgind · 2021-04-26T15:31:29Z

Team triage: closing for inactivity. Happy to reopen if we can work through its issues.

rainersigwald requested changes Jan 4, 2021

View reviewed changes

0xced force-pushed the aliased_cultures branch from bb3815d to ee1b9a3 Compare January 4, 2021 17:13

0xced and others added 5 commits January 4, 2021 21:15

Add support for aliased cultures

03c72ce

Fixes dotnet#3897

Add braces

c96d156

Co-authored-by: Rainer Sigwald <raines@microsoft.com>

Rename InvalidCultureNames to KnownInvalidCultureNames for better cla…

d5f5880

…rity Co-authored-by: Rainer Sigwald <raines@microsoft.com>

Catch CultureNotFoundException instead of Exception

0749c06

0xced force-pushed the aliased_cultures branch from 19c9e11 to 52f3cbb Compare January 4, 2021 20:17

Cleanup FEATURE_CULTUREINFO_GETCULTURE* related code

55f8fee

This was needed in the .NET Core 1.0 era but CultureInfo.GetCultures and CultureInfo.GetCultureInfo are now available in .NET Core (since version 2.0).

0xced mentioned this pull request Jan 4, 2021

Fix AssemblyUtilities.CultureInfoHasGetCultures #6000

Closed

benvillalobos reviewed Jan 5, 2021

View reviewed changes

benvillalobos previously approved these changes Jan 6, 2021

View reviewed changes

Only deem aliased cultures as valid, not _all_ cultures

4bdca75

Check for ThreeLetterISOLanguageName length to be 3 instead of > 0

d68fe2a

On .NET Framework, ThreeLetterISOLanguageName has two letters for unknown cultures such as "xx".

0xced added 6 commits January 6, 2021 15:04

Skip aliased cultures that are not available on .NET Framework

9102445

Skip aliased cultures that are not available on Mono

4388369

Fix handling of invalid cultures with three letters name on .NET Fram…

80bc0f0

…ework

Skip Najdi Arabic (ars) culture on both Mono and .NET Framework

c02b28a

Improve code readability

cdb730c

Fix tests when running under NLS

a10a856

Base automatically changed from master to main March 15, 2021 20:09

Forgind closed this Apr 26, 2021

tarekgh mentioned this pull request May 5, 2021

Cultures aliased by ICU cannot be used for resource localization on non-Windows environments #3897

Closed

	Assert.Equal(new CultureInfo("en-US").ToString(), result);
	result.ShouldBe(new CultureInfo("en-US").ToString());

		@@ -216,6 +256,30 @@ public void PseudoLocalization(string culture)
		Assert.Equal($"MyResource.{culture}.resx", t.AssignedFiles[0].ItemSpec);

	Assert.Equal($"MyResource.{culture}.resx", t.AssignedFiles[0].ItemSpec);
	t.AssignedFiles[0].ItemSpec.ShouldBe($"MyResource.{culture}.resx");

Add support for aliased cultures #6003

Add support for aliased cultures #6003

Conversation

0xced commented Jan 4, 2021

rainersigwald left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tarekgh commented Jan 4, 2021

rainersigwald commented Jan 4, 2021

tarekgh commented Jan 4, 2021

wli3 commented Jan 4, 2021

rainersigwald commented Jan 4, 2021

0xced commented Jan 4, 2021 • edited

tarekgh commented Jan 4, 2021

rainersigwald commented Jan 4, 2021 • edited

0xced commented Jan 4, 2021

rainersigwald commented Jan 4, 2021

Forgind commented Jan 4, 2021 • edited

benvillalobos left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benvillalobos Jan 5, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xced commented Jan 5, 2021

benvillalobos commented Jan 6, 2021 • edited

benvillalobos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xced commented Jan 6, 2021

0xced commented Jan 6, 2021

0xced commented Jan 7, 2021

tarekgh commented Jan 7, 2021 • edited

0xced commented Jan 20, 2021

tarekgh commented Jan 20, 2021

Forgind commented Mar 31, 2021

Forgind commented Apr 26, 2021

0xced commented Jan 4, 2021 •

edited

rainersigwald commented Jan 4, 2021 •

edited

Forgind commented Jan 4, 2021 •

edited

benvillalobos left a comment •

edited

benvillalobos Jan 5, 2021 •

edited

benvillalobos commented Jan 6, 2021 •

edited

tarekgh commented Jan 7, 2021 •

edited