Surfacing arch-specific differences

[sotteson1]

Below is how I'm thinking about solving arch-specific differences. Thoughts, comments?

To solve #384, we talked about using attributes to surface arch-specific differences. I built a 32-bit version of the .winmd and compared it with the normal 64-bit version using WinmdUtils:

dotnet D:\repos\win32metadata\sources\WinmdUtils\bin\Release\netcoreapp3.1\WinmdUtils.dll compare --first D:\repos\win32metadata\bin\Windows.Win32.64.winmd --second D:\repos\win32metadata\bin\Windows.Win32.32.winmd

Some common cases and how I plan to solve them:

• A method or struct in one but not the other: (This one means it's in x64 but not x86)
  Windows.Win32.Debug.Apis.RtlAddFunctionTable(FunctionTable,EntryCount,BaseAddress) not found in 2nd winmd

  Add this to the metadata:
  [ArchDifferences(Supported = Arch.x64)]

• A struct has a different packing on 32-bit: (This one means 64-bit has no packing but 32-bit does)
  Windows.Win32.WindowsAndMessaging.OPENFILENAMEA : => [StructLayout(0,Pack=1)]

  Add this to the metadata:
  [ArchDifferences(32BitPacking = 1)]

• A parameter or return type changes on a method: (This one means the parameter debugApplication changes types between 64 and 32-bit)
  Windows.Win32.Js.Apis.JsCreateContext(runtime,debugApplication,newContext) : debugApplication...IDebugApplication64 => IDebugApplication32

  This one is trickier, but I think the best solution is to emit it twice, once for each signature, and add attributes to indicate which
  one is for which:

  [ArchDifferences(Supported = Arch.x64)]
  [DllImport("chakra", ExactSpelling = true)]
  public unsafe static extern JsErrorCode JsCreateContext([In] void* runtime, [In] IDebugApplication32 debugApplication, [Out] void** newContext);
  
  [ArchDifferences(Supported = Arch.x86)]
  [DllImport("chakra", ExactSpelling = true)]
  public unsafe static extern JsErrorCode JsCreateContext([In] void* runtime, [In] IDebugApplication64 debugApplication, [Out] void** newContext);

  I'm thinking I would do the same thing for a struct where a field was in one or not the other. I could attribute the field, but I could also attribute the entire struct and emit it twice.

• A really annoying one is where lots of methods and structs in Search use a typedef for an int to be either 32 or 64 bits long depending on the arch: (This means it's a 64-bit int vs a 32-bit int)
  Windows.Win32.Search.DBVECTOR.size...UInt64 => UInt32

  Original definition:

  #ifdef _WIN64
  typedef ULONGLONG DBLENGTH;
  #else //_WIN64
  typedef ULONG DBLENGTH;
  #endif
  
  typedef struct tagDBVECTOR {
      DBLENGTH size;
      [size_is((ULONG)size)] void * ptr;
  } DBVECTOR;

  There are a lot of these. The easiest would be to make these UIntPtr or use a NativeTypedef to make a DBLENGTH that wraps a UIntPtr. That would collapse all of these into the same version so I wouldn't need to add any arch-specific attributes. I wish there was a SIZE_T type in ECMA-335.

[AArnott]

What about ARM64? How does that compare?

A parameter or return type changes on a method

The JsCreateContext is an interesting case for two reasons:

1. I can't find any documentation for it.
2. The 64-bit version's type is a COM interface that may derive from the COM interface used in the 32-bit version. If that is true, then we can say that the 64-bit version of the function is always defined since when IDebugApplication64 is passed to the 32-bit function, it'll accept the 64-bit interface since it's a derivative, I think.

Anyway I guess my biggest question on this is how common it is to have types changed on methods across architectures.

I wish there was a SIZE_T type in ECMA-335.

There is: it's IntPtr. And in C# 9, use of nint or nuint actually uses IntPtr underneath (in metadata native int) but attributes its use in APIs so consumers know to prefer nint syntax over IntPtr.
So yes, please find a way to map all these "arch-specific" types that are simply pointer-length integers to IntPtr. Then keeping #21 in mind, also attribute that use of IntPtr (or use a typedef called SIZE_T) to suggest to projections that it should really be a native sized integer, however that is expressed most naturally in the target language.

To not coalesce things like DBLENGTH into one API definition and rather define an API twice with attributes would lead CsWin32 to refuse to emit these APIs until the project targets specific CPU architectures rather than AnyCPU, which would be a real shame since any hand authored p/invoke signature/type would simply use IntPtr/nint in order to retain AnyCPU compatibility.

[kennykerr]

Both IDebugApplication64 and IDebugApplication32 derive from IRemoteDebugApplication.

[kennykerr]

Rust already maps IntPtr/ELEMENT_TYPE_I and UIntPtr/ELEMENT_TYPE_U to Rust's usize and isize types. No need to mint new pointer-sized integer types. You can find these documented in ECMA-335 under the ELEMENT_TYPE constants.

The ArchDifferences attribute seems like a fine solution. I would just keep it simple if possible. So have arch-specific definitions with the ArchDifferences attribute telling us which arch each definition applies to, e.g. [ArchDifferences(x64), but avoid more granular information like packing inside this attribute. There is already metadata for that. Simply encode that in the usual way. Then I would just call the attribute Architecture as it simply identifies a specific architecture that the definition applies to. I think that would just make it a lot simpler to parse.

[kennykerr]

Parsers are also considerably harder to write when there are duplicate type names within a namespace.

[sotteson1]

I would rather attribute individual fields to say if they're only in one arch or the other if we're not going to duplicate struct names.

The main benefit is that the nested struct can carry all of the differences in one place, not only fields, but also things like packing.

Each arch-specific struct (like arch x64, arch x86) would need to contain the entire struct, right? Because you might have 3 common fields, then an x64-only field, then some more common fields, then an x86-only field, more common fields, etc. The CONTEXT struct does this kind of thing.

[kennykerr]

Yes, that's what I would expect.

[AArnott]

I like an approach that doesn't require special checks on each individual field, and doesn't cause us to have to check for the pack size from two distinct attributes. The winning ideas here IMO are an empty struct with the original name but attributes that point to the arch-specific variants (having garbage psuedo-names each), or the nested struct idea.

[kennykerr]

Yes, I don't really care if its nested or not.

[sotteson1]

How does this look? I like the ArchSupported attribute because there are sometimes methods/structs that only appear on certain architectures and then there aren't versions for the other architectures, but the ArchSpecificVersions leads you to the others ones if they exist. I could also combine ArchSupported and ArchSpecific, like: [ArchSupported(Arch.x64 | Arch.arm64, x86 = "JSCreateContext__x86__")]

    [ArchSupported(Arch.x64 | Arch.arm64)]
    [DllImport("USER32", ExactSpelling = true, SetLastError = true)]
    public static extern IntPtr GetWindowLongPtrA([In] HWND hWnd, [In] WINDOW_LONG_PTR_INDEX nIndex);

    [ArchSupported(Arch.x64 | Arch.arm64)]
    [ArchSpecific(x86 = "JSCreateContext__x86__")]
    [DllImport("chakra", ExactSpelling = true)]
    public unsafe static extern JsErrorCode JsCreateContext([In] void* runtime, [In] IDebugApplication64 debugApplication, [Out] void** newContext);

    [ArchSupported(Arch.x86)]
    [DllImport("chakra", EntryPoint = "JSCreateContext")]
    public unsafe static extern JsErrorCode JSCreateContext__x86__([In] void* runtime, [In] IDebugApplication32 debugApplication, [Out] void** newContext);

    [ArchSupported(Arch.x64)]
    [ArchSpecific(x86 = "CONTEXT__x86__", arm64 = "CONTEXT__arm64__")]
    public partial struct CONTEXT
    {
...
    }

    [ArchSupported(Arch.x86)]
    public partial struct CONTEXT__x86__
    {
...
    }

[kennykerr]

Yes, method overloads are supported - WinRT uses them liberally.

Yes, struct overloads seem unusual/unsupported, but Steve seemed to imply that they are possible. And if you can guarantee that there are no ambiguities then the TypeRef to TypeDef mapping would work. All I'm pointing out is that if they are possible then it might be a lot simpler that way.

I'm not sold on any one solution, but some of the alternatives above just seem very complicated.

[sotteson1]

I'm pretty sure I can have duplicate struct names, because that's a bug I've had before that my tests now catch. The problem is that the typerefs that use an arch-specific struct can only point at one of them. GetOpenFileNameW (same signature for all architectures) takes OPENFILENAMEW which has different packing for 64 and 32 bit. So a projection would see the typeref to OPENFILENAMEW and then have to notice that it's arch-specific and then look for the other type with the same name, but with the desired architecture.

[kennykerr]

Ignoring scope, a TypeRef only stores a namespace/name pair so it doesn't point to any specific definition. It merely provides a namespace/name that can be used to look up a TypeDef.

[sotteson1]

Ah, cool! So a projection would have to know there could be more than one struct that matches namespace+name, which is still kind of weird, and then look to match any arch-specific versions. Methods would work the same way.

[AArnott]

Methods would work the same way.

Well, nothing references a method so there's no matching required for that. Except I guess your attributes like RIAA.

I'm still not a fan of deliberately using non-unique type names for reasons I gave above, but I think I can deal with it.