Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building the .NET Core SDK on FreeBSD #1139

Closed
joperator opened this issue Jul 15, 2019 · 220 comments
Closed

Building the .NET Core SDK on FreeBSD #1139

joperator opened this issue Jul 15, 2019 · 220 comments

Comments

@joperator
Copy link

I'm trying to build the .NET Core SDK on FreeBSD but I'm having trouble with the documentation.
The documentation says I just have to initialize the submodules in the source-build repo and run ./build.sh. But this produces only the following output:

Unsupported OS 'FreeBSD' detected. Downloading linux-x64 tools.
Installing dotnet cli...
Installing dotnet 3.0 preview1 cli...
Restoring BuildTools version 3.0.0-preview4-04104-01...

Later in the documentation it says that

git submodule update --init --recursive [...] will initialize the submodules and clone the repos under the src folder if they haven't been initialized.

But after running this command, my src folder contains only the subfolders netcorecli-fsc and reference-assemblies, which are not all required repos I think. As the documentation says, I expected at least core-setup, standard, corefx and coreclr to be cloned into the src folder.
This would also explain why there is no bin folder in my source-build repo after running ./build.sh as described in the documentation.
If the scripts don't work under FreeBSD, I expect at least an error, otherwise I assume that everything has been built, which is not the case here.

So, am I doing something wrong or is the documentation incorrect or out of date?

@wfurt
Copy link
Member

wfurt commented Jul 15, 2019

even source-build needs working SDK and that is problematic since lost build servers. I was planning to revisit the effort but closing on 3.0 prevented me from spending any reasonable time on it.

Aside from that, more cleanup may be needed to deal with OS differences.

@joperator
Copy link
Author

I've already tried to use the old (December 2018) but working .NET Core SDK for FreeBSD as bootstrap CLI as described here. But when running ./init-tools.sh it produces the same output as above and doesn't finish even after several hours.

I could spend some time trying to re-enable .NET Core for FreeBSD, but I need to know where to start because the current documentations don't help me much if they don't work. For example, it would be helpful to know why the build servers were lost and how the .NET Core SDK for FreeBSD that I mentioned was created at that time.

@wfurt
Copy link
Member

wfurt commented Jul 15, 2019

That is something source-build gurus should comment on. Published SDK was built by bootstrapping each needed repo separately and by publishing and consuming intermediate packages. As more repos moved to Arcade, lot of the build process was in flux. I would not be surprised in documentation or code is out of sync.

@crummel
Copy link
Contributor

crummel commented Jul 15, 2019

We've changed our process so that repos are cloned as part of the build process to ./bin/src. This lets us piggyback off other .NET infrastructure to keep source-build up to date more easily. netcorecli-fsc and reference-assemblies are both fake repos that live in source-build, so it's expected you'll have those in ./src. Your error from ./build.sh looks like init-tools is failing - could you post your init-tools.log from the root of the repo?

@joperator
Copy link
Author

Yes, init-tools is indeed failing. The first error was a problem with the SSL certificate while restoring NuGet packages. I could fix it by setting the SSL_CERT_FILE environment variable to /usr/local/share/certs/ca-root-nss.crt as described here. But now there seems to be another problem with the SSL connection:
init-tools.log
The SSL certificate from https://dotnet.myget.org/F/dotnet-buildtools/api/v3/index.json exists in my ca-root-nss.crt file. So I'm surprised that NuGet package restore still seems to have problems establishing an SSL connection.

@joperator
Copy link
Author

It looks like dotnet has problems with the linux-c7-openssl-libs package. I installed it on my FreeBSD machine to ensure full Linux binary compatibility, because the first time I ran ./build.sh I got
No usable version of libssl was found.

After removing this package, reinstalling the default OpenSSL package for FreeBSD, and creating the required symbolic links, running ./init-tools.sh using the .NET Core SDK for FreeBSD as bootstrap CLI succeeded.
I can now tell build.sh to use .NET Core SDK version 3.0.100-preview-010021 via editing DotnetCLIVersion.txt, but running it still fails:
error.txt

@crummel
Copy link
Contributor

crummel commented Jul 17, 2019

Ah, that looks like Darc (the tool we use to clone the source) doesn't build for BSD. I wonder if we can build that portable, I will take a look. Otherwise this is probably a fix in dotnet/arcade-services.

@joperator
Copy link
Author

I was able to build Darc on FreeBSD using target framework netcoreapp3.0 instead of netcoreapp2.1. Cloning the repos now seems to work, but building them still fails. I have run ./build.sh and received the following output:
arcade-error.txt
arcade-build.binlog.txt
arcade.log

I also ran ./build.sh /p:RootRepo=core-setup and received the following output:
core-setup-error.txt
core-setup-build.binlog.txt
linker.log

So in both cases I get the error Segmentation fault (core dumped).
I also receive linux: pid 1234 (dotnet): syscall membarrier not implemented in the terminal, which is not logged into the files.

@crummel
Copy link
Contributor

crummel commented Jul 18, 2019

I've seen similar errors with missing syscalls on ARM. At this point it's probably less of a source-build problem and more getting into bootstrapping the actual SDK on BSD - there's a couple related issues: https://github.com/dotnet/core-setup/issues/5083, dotnet/installer#248, #724.

@joperator
Copy link
Author

Looks like it is a bootstrapping problem, but probably not as you expected @crummel: Although I specified to use the .NET Core SDK for FreeBSD as bootstrap CLI, the build scripts also downloaded the .NET Core SDK 3.0.100-preview5-011568 for Linux and placed it in the source-build/Tools/dotnetcli3 folder. After replacing the folder with source-build/Tools/dotnetcli, which contains the .NET Core SDK for FreeBSD, the build continues.
Now it tries to build Microsoft.DotNet.Arcade.csproj, but fails with error CS1617: Invalid option 'preview' for /langversion. Use '/langversion:?' to list supported values.

@joperator
Copy link
Author

Replacing <langversion>preview</langversion> with <langversion>latest</langversion> in Microsoft.DotNet.Arcade.csproj fixes the above error. It looks like I'm now able to build the main parts of the .NET Core SDK on FreeBSD. I only get the following two errors after the build succeeded:

'common' failed during build.
See '/root/git/dotnet/source-build/bin/logs/common.log' for more information.
'known-good' failed during build.

I've already taken a look at the common.log, but there's nothing there that indicates an error.
So does anyone know what's common and known-good? Are they required for the .NET Core SDK? And where is the .NET Core SDK stored after the build is complete? Or do I have to copy and paste the parts manually?
Any help would be appreciated. Thanks in advance.

@omajid
Copy link
Member

omajid commented Jul 31, 2019

AFAIK known-good is the set of components that need to be built to assemble .NET Core itself. Right now, it's roughly analogous to the all target in other projects' makefiles.

If everything (including the cli) built succefully, the result should be a bin/x64/Release/dotnet-sdk-*tar.gz file. That's the .NET Core SDK itself. For fedora that's pretty much all we need. We extract it to /usr/lib64/dotnet.

@wfurt
Copy link
Member

wfurt commented Jul 31, 2019

That is great progress @joperator. Do you have any outstanding changes or just something you need to set?
With 3.0 under control now, I should have some cycles again to give FreeBSD another try.
Just to confirm: You use FreeBSD bootstrap cli, not Linux under emulation, right?

@joperator
Copy link
Author

@omajid: Then it looks like I have to build known-good as well because my bin/x64/Release folder only contains an empty netcorecli-fsc directory.

@wfurt: There are outstanding changes as well as some things you need to set to successfully run build.sh on FreeBSD. But these changes are only limited to a few files, where some of them are already described above. I will soon provide a complete list of these necessary changes.
I'm using FreeBSD bootstrap CLI to build the .NET projects. While running build.sh it downloaded and tried to use the .NET Core SDK for Linux. But that didn't work, as I described above.

@wfurt
Copy link
Member

wfurt commented Aug 1, 2019

Very good @joperator. I'm hoping to get newer version of the SDK published (closer to 3.0 final content) but I could not build some components again. If you get build working it would help me with that effort.

As far as the Linux SDK: I made changes in several components to recognize FreeBSD properly. I was on Linux path for a while until I realized that is dead-end path. Even if the binaries work under emulation it still uses many files from /proc and that will not work on FreeBSD. Getting source build working is best long-term option for FreeBSD. I would still like to build components using published sdk to avoid regressions.

@joperator
Copy link
Author

I agree with you @wfurt:

Getting source build working is best long-term option for FreeBSD.

Therefore, here is a complete list of the necessary settings and changes to build the source-build repo on FreeBSD.
I have built it using the current master (Aug 2, 2019) of the source-build repo and a VM with FreeBSD 12.0-RELEASE amd64 on a Windows 10 host.

Install and configure required packages

Install git

pkg install git

Install libunwind

pkg install libunwind

Install and configure bash (Building-.NET-Core-3.x-on-FreeBSD)

pkg install bash
ln -s /usr/local/bin/bash /bin/bash

Install and configure OpenSSL (dotnet/coreclr#22124)

pkg install openssl
ln -s /usr/local/lib/libssl.so.9 /usr/local/lib/libssl.so.8
ln -s /usr/local/lib/libcrypto.so.9 /usr/local/lib/libcrypto.so.8

Clone the source-build repo

git clone https://github.com/dotnet/source-build.git ~/git/dotnet/source-build

Set your environment variables

Prevent the dotnet CLI from freezing with no output (dotnet/coreclr#22124)

setenv LD_PRELOAD /usr/lib/libpthread.so

Set the path of the bootstrap CLI (Use the bootstrap CLI to build source-build). I have used dotnet-sdk-latest-freebsd-x64.tar.gz as bootstrap CLI.

setenv DotNetBootstrapCliTarPath $(DotNetBootstrapCliTarDir)/dotnet.tar

Build the source-build repo

cd ~/git/dotnet/source-build

./init-tools.sh
  ...
  Initializing BuildTools...
  ...
  Done initializing BuildTools.
  Initializing Arcade...
  Done initializing Arcade.
  Done initializing tools.

./build.sh
  ...
  warning: submodules tools-local/arcade-services does not appear to be initialized.
  Should I initialize it for you? [Y]es / [n]o / [a]ll / [q]uit] y
  ...
  Tools are already initialized
  ...
  No executable found matching command "dotnet-/root/git/dotnet/source-build/Tools/dotnetcli/sdk/2.2.300/MSBuild.dll". See https://aka.ms/missing-command for more information.

Fix dotnet CLI version

ee DotnetCLIVersion.txt

Replace 2.2.300 with 3.0.100-preview-010021.

Replace the downloaded dotnet CLI for Linux with the bootstrap CLI for FreeBSD

rm -r Tools/dotnetcli3
cp -r Tools/dotnetcli Tools/dotnetcli3

./build.sh
  ...
  /root/git/dotnet/source-build/tools-local/init-build.proj(153,5): error MSB3073: The command "/root/git/dotnet/source-build/Tools/dotnetcli/dotnet build -c Release /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Microsoft.DotNet.Darc.csproj" exited with code 1.

Change target framework netcoreapp2.1 to netcoreapp3.0 and commit your changes

cd tools-local/arcade-services
ee src/Microsoft.DotNet.Darc/src/Darc/Microsoft.DotNet.Darc.csproj

Replace <TargetFramework>netcoreapp2.1</TargetFramework> with <TargetFramework>netcoreapp3.0</TargetFramework>.

git commit -a -m "Change target framework of 'Microsoft.DotNet.Darc.csproj' from 'netcoreapp2.1' to 'netcoreapp3.0'."
cd ../..
git add tools-local/arcade-services
git commit -m "Update submodule 'tools-local/arcade-services'."

./build.sh
  ...
  No executable found matching command "dotnet-/root/git/dotnet/source-build/tools-local/arcade-services/artifacts/bin/Microsoft.DotNet.Darc/Release/netcoreapp2.1/Microsoft.DotNet.Darc.dll". See https://aka.ms/missing-command for more information.
  /root/git/dotnet/source-build/tools-local/init-build.proj(180,5): error MSB3073: The command "/root/git/dotnet/source-build/Tools/dotnetcli/dotnet /root/git/dotnet/source-build/tools-local/arcade-services/artifacts/bin/Microsoft.DotNet.Darc/Release/netcoreapp2.1/Microsoft.DotNet.Darc.dll clone --repos-folder /root/git/dotnet/source-build/bin/src/ --git-dir-folder /root/git/dotnet/source-build/.git/modules/src/ --include-toolset --ignore-repos ";https://dev.azure.com/dnceng/internal/_git/dotnet-optimization;https://dev.azure.com/devdiv/DevDiv/_git/DotNet-Trusted;https://devdiv.visualstudio.com/DevDiv/_git/DotNet-Trusted;https://dnceng@dev.azure.com/dnceng/internal/_git/dotnet-optimization;https://dev.azure.com/dnceng/internal/_git/dotnet-core-setup" --azdev-pat bogus --github-pat bogus --depth 0 --debug" exited with code 1.

Copy the binaries to the expected directory because we changed the target framework from netcoreapp2.1 to netcoreapp3.0

rm -r tools-local/arcade-services/artifacts/bin/Microsoft.DotNet.Darc/Release/netcoreapp2.1
cp -r tools-local/arcade-services/artifacts/bin/Microsoft.DotNet.Darc/Release/netcoreapp3.0 tools-local/arcade-services/artifacts/bin/Microsoft.DotNet.Darc/Release/netcoreapp2.1

./build.sh
  ...
          Something failed while cloning.
    System.Exception: Something went wrong when cloning repo https://github.com/dotnet/arcade at <default branch> into /root/git/dotnet/source-build/bin/src/arcade ---> System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. ---> System.DllNotFoundException: Unable to load shared library 'git2-572e4d8' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Shared object "libgit2-572e4d8" not found, required by "dotnet"
       at LibGit2Sharp.Core.NativeMethods.git_libgit2_init()
       at LibGit2Sharp.Core.NativeMethods.InitializeNativeLibrary()
       at LibGit2Sharp.Core.NativeMethods..cctor()
       --- End of inner exception stack trace ---
       at LibGit2Sharp.Core.NativeMethods.git_clone(git_repository*& repo, String origin_url, FilePath workdir_path, GitCloneOptions& opts)
       at LibGit2Sharp.Core.Proxy.git_clone(String url, String workdir, GitCloneOptions& opts)
       at LibGit2Sharp.Repository.Clone(String sourceUrl, String workdirPath, CloneOptions options)
       at Microsoft.DotNet.DarcLib.RemoteRepoBase.Clone(String repoUri, String commit, String targetDirectory, ILogger _logger, String pat, String gitDirectory) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/RemoteRepoBase.cs:line 176
       --- End of inner exception stack trace ---
       at Microsoft.DotNet.DarcLib.RemoteRepoBase.Clone(String repoUri, String commit, String targetDirectory, ILogger _logger, String pat, String gitDirectory) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/RemoteRepoBase.cs:line 210
       at Microsoft.DotNet.DarcLib.GitHubClient.Clone(String repoUri, String commit, String targetDirectory, String gitDirectory) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/GitHubClient.cs:line 875
       at Microsoft.DotNet.DarcLib.Remote.Clone(String repoUri, String commit, String targetDirectory, String gitDirectory) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/Actions/Remote.cs:line 983
       at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopyAndCreateGitDir(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, String gitDirRedirect, ILogger log) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 284
       at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopyWithGitDirPath(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, ILogger log) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 272
       at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopy(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, ILogger log) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 225
       at Microsoft.DotNet.Darc.Operations.CloneOperation.ExecuteAsync() in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 102
  /root/git/dotnet/source-build/tools-local/init-build.proj(180,5): error MSB3073: The command "/root/git/dotnet/source-build/Tools/dotnetcli/dotnet /root/git/dotnet/source-build/tools-local/arcade-services/artifacts/bin/Microsoft.DotNet.Darc/Release/netcoreapp2.1/Microsoft.DotNet.Darc.dll clone --repos-folder /root/git/dotnet/source-build/bin/src/ --git-dir-folder /root/git/dotnet/source-build/.git/modules/src/ --include-toolset --ignore-repos ";https://dev.azure.com/dnceng/internal/_git/dotnet-optimization;https://dev.azure.com/devdiv/DevDiv/_git/DotNet-Trusted;https://devdiv.visualstudio.com/DevDiv/_git/DotNet-Trusted;https://dnceng@dev.azure.com/dnceng/internal/_git/dotnet-optimization;https://dev.azure.com/dnceng/internal/_git/dotnet-core-setup" --azdev-pat bogus --github-pat bogus --depth 0 --debug" exited with code 42.

Install missing libraries for LibGit2Sharp

pkg install libgit2
ln -s /usr/local/lib/libgit2.so /usr/local/lib/libgit2-572e4d8.so

./build.sh
  ...
          Something failed while cloning.
    System.Exception: Something went wrong when cloning repo https://github.com/mono/linker.git at refs/heads/master into /root/git/dotnet/source-build/bin/src/linker ---> LibGit2Sharp.LibGit2SharpException: failed to connect to github.com: Operation timed out
       at LibGit2Sharp.Core.Ensure.HandleError(Int32 result)
       at LibGit2Sharp.Core.Proxy.git_submodule_update(SubmoduleHandle submodule, Boolean init, GitSubmoduleUpdateOptions& options)
       at LibGit2Sharp.SubmoduleCollection.Update(String name, SubmoduleUpdateOptions options)
       at Microsoft.DotNet.DarcLib.RemoteRepoBase.CheckoutSubmodules(Repository repo, CloneOptions submoduleCloneOptions, String gitDirParentPath, ILogger log) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/RemoteRepoBase.cs:line 222
       at Microsoft.DotNet.DarcLib.RemoteRepoBase.Clone(String repoUri, String commit, String targetDirectory, ILogger _logger, String pat, String gitDirectory) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/RemoteRepoBase.cs:line 205
       --- End of inner exception stack trace ---
       at Microsoft.DotNet.DarcLib.RemoteRepoBase.Clone(String repoUri, String commit, String targetDirectory, ILogger _logger, String pat, String gitDirectory) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/RemoteRepoBase.cs:line 210
       at Microsoft.DotNet.DarcLib.GitHubClient.Clone(String repoUri, String commit, String targetDirectory, String gitDirectory) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/GitHubClient.cs:line 875
       at Microsoft.DotNet.DarcLib.Remote.Clone(String repoUri, String commit, String targetDirectory, String gitDirectory) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/Actions/Remote.cs:line 983
       at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopyAndCreateGitDir(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, String gitDirRedirect, ILogger log) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 284
       at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopyWithGitDirPath(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, ILogger log) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 272
       at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopy(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, ILogger log) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 225
       at Microsoft.DotNet.Darc.Operations.CloneOperation.ExecuteAsync() in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 102
  /root/git/dotnet/source-build/tools-local/init-build.proj(180,5): error MSB3073: The command "/root/git/dotnet/source-build/Tools/dotnetcli/dotnet /root/git/dotnet/source-build/tools-local/arcade-services/artifacts/bin/Microsoft.DotNet.Darc/Release/netcoreapp2.1/Microsoft.DotNet.Darc.dll clone --repos-folder /root/git/dotnet/source-build/bin/src/ --git-dir-folder /root/git/dotnet/source-build/.git/modules/src/ --include-toolset --ignore-repos ";https://dev.azure.com/dnceng/internal/_git/dotnet-optimization;https://dev.azure.com/devdiv/DevDiv/_git/DotNet-Trusted;https://devdiv.visualstudio.com/DevDiv/_git/DotNet-Trusted;https://dnceng@dev.azure.com/dnceng/internal/_git/dotnet-optimization;https://dev.azure.com/dnceng/internal/_git/dotnet-core-setup" --azdev-pat bogus --github-pat bogus --depth 0 --debug" exited with code 42.

./build.sh
  ...
          Something failed while cloning.
    System.Exception: Something went wrong when checking out $(GitCommitHash) in /root/git/dotnet/source-build/bin/src/linker.$(GitCommitHash) ---> System.InvalidOperationException: Sequence contains no matching element
       at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source, Func`2 predicate)
       at Microsoft.DotNet.DarcLib.LocalGitClient.CleanRepoAndSubmodules(Repository repo, ILogger log) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/LocalGitClient.cs:line 427
       at Microsoft.DotNet.DarcLib.LocalGitClient.Checkout(String repoDir, String commit, Boolean force) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/LocalGitClient.cs:line 308
       --- End of inner exception stack trace ---
       at Microsoft.DotNet.DarcLib.LocalGitClient.Checkout(String repoDir, String commit, Boolean force) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/LocalGitClient.cs:line 347
       at Microsoft.DotNet.DarcLib.Local.Checkout(String commit, Boolean force) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/Actions/Local.cs:line 144
       at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleRepoAtSpecificHash(String repoPath, String commit, String masterRepoGitDirPath, ILogger log) in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 218
       at Microsoft.DotNet.Darc.Operations.CloneOperation.ExecuteAsync() in /root/git/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 106
  /root/git/dotnet/source-build/tools-local/init-build.proj(180,5): error MSB3073: The command "/root/git/dotnet/source-build/Tools/dotnetcli/dotnet /root/git/dotnet/source-build/tools-local/arcade-services/artifacts/bin/Microsoft.DotNet.Darc/Release/netcoreapp2.1/Microsoft.DotNet.Darc.dll clone --repos-folder /root/git/dotnet/source-build/bin/src/ --git-dir-folder /root/git/dotnet/source-build/.git/modules/src/ --include-toolset --ignore-repos ";https://dev.azure.com/dnceng/internal/_git/dotnet-optimization;https://dev.azure.com/devdiv/DevDiv/_git/DotNet-Trusted;https://devdiv.visualstudio.com/DevDiv/_git/DotNet-Trusted;https://dnceng@dev.azure.com/dnceng/internal/_git/dotnet-optimization;https://dev.azure.com/dnceng/internal/_git/dotnet-core-setup" --azdev-pat bogus --github-pat bogus --depth 0 --debug" exited with code 42.

Clone jbevain/cecil to the expected directory

rm -r bin/src/linker.$(GitCommitHash)/external/cecil
git clone https://github.com/jbevain/cecil.git bin/src/linker.$(GitCommitHash)/external/cecil

./build.sh
  ...
  /root/git/dotnet/source-build/repos/dir.targets(281,5): error MSB3073: The command "/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/build.sh  --pack --configuration Release --binaryLog  /p:DotNetPackageVersionPropsPath=/root/git/dotnet/source-build/bin/obj/x64/Release/PackageVersions.props /p:DotNetRestoreSourcePropsPath=/root/git/dotnet/source-build/bin/obj/x64/Release/RestoreSources.props >> /root/git/dotnet/source-build/bin/logs/sourcelink.log 2>&1" exited with code 145. [/root/git/dotnet/source-build/repos/sourcelink.proj]
    Found dotnet SDK, but did not find dotnet.dll at [/root/git/dotnet/source-build/Tools/dotnetcli3/sdk/3.0.100-preview6-012264/dotnet.dll]
    Build failed (exit code '145').

    'sourcelink' failed during build.
    See '/root/git/dotnet/source-build/bin/logs/sourcelink.log' for more information.
    'known-good' failed during build.

Copy the dotnet SDK to the expected directory because we replaced the dotnet CLI for Linux with the bootstrap CLI for FreeBSD

cp -r Tools/dotnetcli3/sdk/3.0.100-preview-010021/* Tools/dotnetcli3/sdk/3.0.100-preview6-012264

./build.sh
  ...
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.GitHub/Microsoft.SourceLink.GitHub.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.Common/Microsoft.SourceLink.Common.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.Vsts.Git/Microsoft.SourceLink.Vsts.Git.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/Microsoft.Build.Tasks.Git/Microsoft.Build.Tasks.Git.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.GitHub/Microsoft.SourceLink.GitHub.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.Common/Microsoft.SourceLink.Common.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/Microsoft.Build.Tasks.Git/Microsoft.Build.Tasks.Git.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.Vsts.Git/Microsoft.SourceLink.Vsts.Git.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.Vsts.Tfvc/Microsoft.SourceLink.Vsts.Tfvc.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.GitLab/Microsoft.SourceLink.GitLab.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.Vsts.Tfvc/Microsoft.SourceLink.Vsts.Tfvc.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.Bitbucket.Git/Microsoft.SourceLink.Bitbucket.Git.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.GitLab/Microsoft.SourceLink.GitLab.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.Tfs.Git/Microsoft.SourceLink.Tfs.Git.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.Bitbucket.Git/Microsoft.SourceLink.Bitbucket.Git.csproj]
    CSC : error CS1617: Invalid option 'Preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/sourcelink.$(GitCommitHash)/src/SourceLink.Tfs.Git/Microsoft.SourceLink.Tfs.Git.csproj]
        0 Warning(s)
        16 Error(s)

    Time Elapsed 00:00:14.68
    Build failed (exit code '1').

    'sourcelink' failed during build.
    See '/root/git/dotnet/source-build/bin/logs/sourcelink.log' for more information.
    'known-good' failed during build.

Replace language version Preview with 8.0

ee bin/src/sourcelink.$(GitCommitHash)/src/Directory.Build.props

Replace <LangVersion>Preview</LangVersion> with <LangVersion>8.0</LangVersion>.

Remove modifier readonly from properties because it is not valid for these items

ee bin/src/sourcelink.$(GitCommitHash)/src/Microsoft.Build.Tasks.Git/GitDataReader/GitRepositoryLocation.cs

Remove modifier readonly from properties.

./build.sh
  ...
    CSC : error CS1617: Invalid option 'preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/src/Microsoft.DotNet.Arcade.Sdk/Microsoft.DotNet.Arcade.Sdk.csproj]
      Microsoft.DotNet.SignTool -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.SignTool/Release/netcoreapp2.0/Microsoft.DotNet.SignTool.dll
      Microsoft.DotNet.Deployment.Tasks.Links -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.Deployment.Tasks.Links/Release/net472/Microsoft.DotNet.Deployment.Tasks.Links.dll
      Microsoft.DotNet.Deployment.Tasks.Links -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.Deployment.Tasks.Links/Release/netcoreapp2.0/Microsoft.DotNet.Deployment.Tasks.Links.dll
      Microsoft.DotNet.GenFacades -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.GenFacades/Release/net472/Microsoft.DotNet.GenFacades.dll
    CSC : error CS1617: Invalid option 'preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/src/Microsoft.DotNet.Arcade.Sdk/Microsoft.DotNet.Arcade.Sdk.csproj]
      Microsoft.DotNet.GenFacades -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.GenFacades/Release/netcoreapp2.1/Microsoft.DotNet.GenFacades.dll
      Microsoft.DotNet.CoreFxTesting -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.CoreFxTesting/Release/netcoreapp2.1/Microsoft.DotNet.CoreFxTesting.dll
      Microsoft.DotNet.CoreFxTesting -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.CoreFxTesting/Release/net472/Microsoft.DotNet.CoreFxTesting.dll
      Microsoft.DotNet.Build.Tasks.Packaging -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.Build.Tasks.Packaging/Release/net472/Microsoft.DotNet.Build.Tasks.Packaging.dll
      Microsoft.DotNet.Build.Tasks.Packaging -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.Build.Tasks.Packaging/Release/netcoreapp2.1/Microsoft.DotNet.Build.Tasks.Packaging.dll
      Microsoft.DotNet.Build.Tasks.Configuration -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.Build.Tasks.Configuration/Release/netcoreapp2.1/Microsoft.DotNet.Build.Tasks.Configuration.dll
      Microsoft.DotNet.Build.Tasks.Configuration -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.Build.Tasks.Configuration/Release/net472/Microsoft.DotNet.Build.Tasks.Configuration.dll
      Microsoft.DotNet.CodeAnalysis -> /root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/artifacts/bin/Microsoft.DotNet.CodeAnalysis/Release/netstandard2.0/Microsoft.DotNet.CodeAnalysis.dll

    Build FAILED.

    CSC : error CS1617: Invalid option 'preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/src/Microsoft.DotNet.Arcade.Sdk/Microsoft.DotNet.Arcade.Sdk.csproj]
    CSC : error CS1617: Invalid option 'preview' for /langversion. Use '/langversion:?' to list supported values. [/root/git/dotnet/source-build/bin/src/arcade.$(GitCommitHash)/src/Microsoft.DotNet.Arcade.Sdk/Microsoft.DotNet.Arcade.Sdk.csproj]
        0 Warning(s)
        2 Error(s)

    Time Elapsed 00:00:24.02
    Build failed (exit code '1').

    'arcade' failed during build.
    See '/root/git/dotnet/source-build/bin/logs/arcade.log' for more information.
    'known-good' failed during build.

Replace language version preview with latest

ee bin/src/arcade.$(GitCommitHash)/src/Microsoft.DotNet.Arcade.Sdk/Microsoft.DotNet.Arcade.Sdk.csproj

Replace <LangVersion>preview</LangVersion> with <LangVersion>latest</LangVersion>.

Fix false positives

If you get messages like

/root/git/dotnet/source-build/repos/dir.targets(378,5): error : $(Repo) produced no new source-built package identities. Known possible causes: [/root/git/dotnet/source-build/repos/$(Repo).proj]
/root/git/dotnet/source-build/repos/dir.targets(378,5): error : ^ There may have been a silent failure in the submodule build. To confirm, check the build log file for undetected errors that may have prevented package creation: /root/git/dotnet/source-build/bin/logs/$(Repo).log [/root/git/dotnet/source-build/repos/$(Repo).proj]
/root/git/dotnet/source-build/repos/dir.targets(378,5): error : ^ This error might be a false positive if $(Repo) intentionally builds no nuget packages. If so, set the SkipEnsurePackagesCreated property to true in /root/git/dotnet/source-build/repos/$(Repo).proj [/root/git/dotnet/source-build/repos/$(Repo).proj]
/root/git/dotnet/source-build/repos/dir.targets(378,5): error : ^ The 'bin' directory might be dirty from a previous build and the package files already existed. If so, perform a clean build, or check which packages were already in 'bin' by opening /root/git/dotnet/source-build/bin/obj/x64/Release/PackageVersions.props.pre.$(Repo).xml [/root/git/dotnet/source-build/repos/$(Repo).proj]
/root/git/dotnet/source-build/repos/dir.targets(378,5): error : ^ The packages may have been written to an unexpected directory. For example, some repos used bin/ and changed to artifacts/ to match Arcade. Check PackagesOutput in /root/git/dotnet/source-build/repos/$(Repo).proj (currently '/root/git/dotnet/source-build/bin/src/$(Repo).$(GitCommitHash)/artifacts/packages/Release/NonShipping/') [/root/git/dotnet/source-build/repos/$(Repo).proj]

then make sure that there has not been a silent failure in the submodule builds, set the SkipEnsurePackagesCreated property as described and run ./build.sh again.

Fix missing app host freebsd-x64

If you get a message like

/root/git/dotnet/source-build/Tools/dotnetcli/sdk/3.0.100-preview-010021/Sdks/Microsoft.NET.Sdk/targets/Microsoft.PackageDependencyResolution.targets(220,5): error NETSDK1065: Cannot find app host for freebsd-x64. freebsd-x64 could be an invalid runtime identifier (RID). For more information about RID, see https://aka.ms/rid-catalog.

then just run ./build.sh again and the error will suddenly disappear.

Build completed successfully

After the build completed you should see the following output:

Executing: dotnet pack
  Successfully created package '/root/git/dotnet/source-build/bin/src/common.$(GitCommitHash)/artifacts/build/Microsoft.Extensions.CommandLineUtils.1.0.1.nupkg'.
Skipping tests because -NoTest was specified

Done


'common' failed during build.
See 'root/git/dotnet/source-build/bin/logs/common.log' for more information.
'known-good' failed during build.

A look at bin/logs/common.log indicates that common does not fail during build.
If known-good is necessary to assemble the .NET Core SDK itself as described above, it looks like it really fails during build, because my bin/x64/Release folder only contains an empty netcorecli-fsc directory. It would be great if someone could figure out why known-good fails during build, because there is neither an error in the terminal nor a log file.

@crummel
Copy link
Contributor

crummel commented Aug 2, 2019

Known-good will fail if anything before it does, so that's probably what's happening. Could I get your whole bin/logs directory? And any *.binlog files under the repo root would be helpful, they get scattered in the individual bin/src directories.

@joperator
Copy link
Author

Yes, of course. Here are the logs from the bin/logs directory:
logs.zip

I had to split the *.binlog files into several zip archives so that each of them is smaller than the 10MB upload limit. The attached *.binlog files were located in these directories:

~/git/dotnet/source-build/src/netcorecli-fsc/msbuild.binlog
~/git/dotnet/source-build/build.binlog
~/git/dotnet/source-build/bin/src/arcade.8f3c22397990aeb20a88690b51dad4b33f21e7ff/artifacts/log/Release/ToolsetRestore.binlog
~/git/dotnet/source-build/bin/src/arcade.8f3c22397990aeb20a88690b51dad4b33f21e7ff/artifacts/log/Release/Build.binlog
~/git/dotnet/source-build/bin/src/sourcelink.51310e65e75010467993f793e1739d1a1dad50c5/artifacts/log/Release/ToolsetRestore.binlog
~/git/dotnet/source-build/bin/src/sourcelink.51310e65e75010467993f793e1739d1a1dad50c5/artifacts/log/Release/Build.binlog
~/git/dotnet/source-build/bin/src/xdt.c01a538851a8ab1a1fbeb2e6243f391fff7587b4/artifacts/log/Release/Build.binlog
~/git/dotnet/source-build/bin/src/standard.a9422da6ed91b413019ef93051613c5f03801d64/artifacts/log/Release/Build.binlog
~/git/dotnet/source-build/bin/src/roslyn.c91adff42c488aef2c2c532a7b053fb55e0c16ea/SourceBuild.binlog
~/git/dotnet/source-build/bin/src/roslyn.c91adff42c488aef2c2c532a7b053fb55e0c16ea/Microsoft.NETCore.Compilers.Package.binlog
~/git/dotnet/source-build/bin/src/cliCommandLineParser.0e89c2116ad28e404ba56c14d1c3f938caa25a01/artifacts/log/Release/Build.binlog
~/git/dotnet/source-build/bin/src/ApplicationInsights-dotnet.53b80940842204f78708a538628288ff5d741a1d/restore.binlog
~/git/dotnet/source-build/bin/src/ApplicationInsights-dotnet.53b80940842204f78708a538628288ff5d741a1d/build.binlog
~/git/dotnet/source-build/bin/src/ApplicationInsights-dotnet.53b80940842204f78708a538628288ff5d741a1d/pack.binlog
~/git/dotnet/source-build/bin/src/xliff-tasks.173ee3bd61c9549557eefa3cfb718bdef157cb87/msbuild.binlog
~/git/dotnet/source-build/bin/src/Newtonsoft.Json.cac0690ad133c5e166ce5642dc71175791404fad/restore.binlog
~/git/dotnet/source-build/bin/src/Newtonsoft.Json.cac0690ad133c5e166ce5642dc71175791404fad/build.binlog
~/git/dotnet/source-build/bin/src/Newtonsoft.Json.cac0690ad133c5e166ce5642dc71175791404fad/pack.binlog

And here are the attached *.binlog files:
netcorecli-fsc.zip
source-build.zip
arcade.8f3c22397990aeb20a88690b51dad4b33f21e7ff.zip
sourcelink.51310e65e75010467993f793e1739d1a1dad50c5.zip
xdt.c01a538851a8ab1a1fbeb2e6243f391fff7587b4.zip
standard.a9422da6ed91b413019ef93051613c5f03801d64.zip
roslyn.c91adff42c488aef2c2c532a7b053fb55e0c16ea.zip
cliCommandLineParser.0e89c2116ad28e404ba56c14d1c3f938caa25a01.zip
ApplicationInsights-dotnet.53b80940842204f78708a538628288ff5d741a1d.zip
xliff-tasks.173ee3bd61c9549557eefa3cfb718bdef157cb87.zip
Newtonsoft.Json.cac0690ad133c5e166ce5642dc71175791404fad.zip

@crummel
Copy link
Contributor

crummel commented Aug 5, 2019

It looks like Roslyn is failing, possibly silently if you're not seeing it:
error NETSDK1065: Cannot find app host for freebsd-x64. freebsd-x64 could be an invalid runtime identifier (RID). We don't have a declared dependency from common on Roslyn, so I'm not sure why that's coming up as the source repo, but I think it might have to do with rebuilding - incremental builds don't work very well for us right now (which is also why you're getting the no packages produced false positives), so we recommend doing a git clean -fxd between builds. I'm not sure if the RID error means we need to patch it in somewhere or if we're not picking up the BSD runtime correctly - any ideas @wfurt?

@wfurt
Copy link
Member

wfurt commented Aug 5, 2019

We did not build Roslyn in official builds as that is same on all platforms and comes in from NuGet package. We will probably need to submit fixes there to recognize FreeBSD as platform.

BTW is there option @crummel to build just runtime instead of whole SDK? I know we want SDK eventually but getting at least runtime would be good milestone.

@joperator
Copy link
Author

I can build SourceBuild.sln from the dotnet/roslyn repo after applying the following patches:

  • In roslyn/eng/targets/Settings.props add netcoreapp3.0 to the RoslynPortableTargetFrameworks property and freebsd-x64 to the RoslynPortableRuntimeIdentifiers property.
  • In roslyn/src/Compilers/Core/MSBuildTask/Microsoft.BuildTasks.CodeAnalysis.csproj replace
<PackageReference Include="System.IO.Pipes.AccessControl" Version="$(SystemIOPipesAccessControlVersion)" Condition="'$(TargetFramework)' == 'netcoreapp2.1'" />
  with
<PackageReference Include="System.IO.Pipes.AccessControl" Version="$(SystemIOPipesAccessControlVersion)" Condition="'$(TargetFramework)' == 'netcoreapp2.1' Or '$(TargetFramework)' == 'netcoreapp3.0'" />
  • In roslyn/src/Compilers/VisualBasic/Test/Emit/Microsoft.CodeAnalysis.VisualBasic.Emit.UnitTests.vbproj add
<DefineConstants Condition="'$(TargetFramework)' == 'netcoreapp3.0'">$(FinalDefineConstants),NETCOREAPP3_0=-1</DefineConstants>
  • In the *.cs files adjust the preprocessor directives containing NETCOREAPP2_1. For example:
    • Change #if NET472 || NETCOREAPP2_1 to #if NET472 || NETCOREAPP2_1 || NETCOREAPP3_0
    • Change #if !NETCOREAPP2_1 to #if !NETCOREAPP2_1 && !NETCOREAPP3_0

./build.sh from dotnet/source-build tries to build the Roslyn projects the same way, but as you've already mentioned @crummel, the incremental builds don't work very well and don't allow me to build the patched Roslyn projects, because after ./build.sh has cloned and failed to build the dotnet/roslyn repo, it won't try to rebuild the repo again when running ./build.sh the next time. The easiest way to fix building the Roslyn projects on FreeBSD, would be to send the above patches to dotnet/roslyn, I think.

@wfurt
Copy link
Member

wfurt commented Aug 6, 2019

that would make sense. Why don't you create PR @joperator and mention me as well. Master branches should be now open for 5.0 work.

@crummel
Copy link
Contributor

crummel commented Aug 6, 2019

BTW is there option @crummel Chris Rummel FTE to build just runtime instead of whole SDK?

We can add /p:RootRepo=core-setup to the ./build.sh command, but this will still build Roslyn because the CoreFX build requires a more recent C# compiler than is in the preview6 SDK, as well as to support coherency.

@crummel
Copy link
Contributor

crummel commented Aug 6, 2019

after ./build.sh has cloned and failed to build the dotnet/roslyn repo, it won't try to rebuild the repo again when running ./build.sh the next time.

Yeah, our process here isn't great and it's why we're working to drive down the number of patches. Currently I go into the bin/src/<repo> directory (not bin/src/<repo>.<hash> as it's not an actual git directory), git am ../../../patches/<repo>/*, make my new change and commit it, git format-patch -o ../../../patches/<repo> <hash>, then git add the new patch, clean, and build.

@joperator
Copy link
Author

It's more complicated than I thought @wfurt:
Creating a PR with the above patches for the dotnet/roslyn repo would probably fix the FreeBSD build, but it would break the build for Windows and other platforms. The problem is that targeting netcoreapp2.1 and netcoreapp3.0 side-by-side is not possible everywhere in the project. So I would have to replace netcoreapp2.1 with netcoreapp3.0 in the whole project, which is a big change instead of a small patch. But I think it's going to be changed to netcoreapp3.0 in the future anyway. So maybe I can speed up this process a little bit...

@wfurt
Copy link
Member

wfurt commented Aug 7, 2019

That make sense. I bump to something similar while back with coreclr. I can ask around what the plan is. As 3.0 is almost done, I would expect we would use it for building at some point.

@daydrim
Copy link

daydrim commented Aug 11, 2019

Hello!

I can get init-tools.sh work on FreeBSD 12 p9 Azure with DotNet3CLIVersion 3.0.100-preview7-012821 and BuildTools 3.0.0-preview7-27912-14
Additionally use the instructions above.

kldload linux.ko
kldload linux64

@daydrim
Copy link

daydrim commented Aug 11, 2019

Hello!
I can get init-tools.sh work on FreeBSD 12 p9 Azure with DotNet3CLIVersion 3.0.100-preview7-012821 and BuildTools 3.0.0-preview4-04212-02
Additionally use the instructions above.
kldload linux.ko
kldload linux64

I've stuck on step

Install missing libraries for LibGit2Sharp

pkg install libgit2

/root/dot-net/source-build/packages/microsoft.dotnet.arcade.sdk/1.0.0-dev.18424.13/tools/Build.proj(63,5): error : Property 'Projects' must be specified

Build FAILED.

/root/dot-net/source-build/packages/microsoft.dotnet.arcade.sdk/1.0.0-dev.18424.13/tools/Build.proj(63,5): error : Property 'Projects' must be specified
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:00.57
Build failed (exit code '1').

'arcade' failed during build.
See '/root/dot-net/source-build/bin/logs/arcade.log' for more information.
'known-good' failed during build.

@wfurt
Copy link
Member

wfurt commented Aug 11, 2019

running Linux runtime under emulation may work for simple app but I think it is dead end. It is not only about binaries but also some managed code is built from different sources. For example process management depends and some other functions do depend on Linux proc and that is incomplete under emulation.

Also note, that you cannot load FreeBSD libraries to Linux process. Using libgit2 from ports will not work wIth Linux SDK. You would need to get copy from Linux.

After all the long threads I'm wondering if we need to assemble new bootstrap cli. We may be able to keep runtime as it but drop new skd on top of it.

@nkosi23
Copy link

nkosi23 commented Mar 19, 2022

In parallel to the present approach we are pursuing, does anyone know top off the mind what is the first .NET Core release that could be built without an SDK and where build instructions can be found? Maybe this rabbit hole will turn out to be somehow easier and more sustainable than setting up and maintaining these cross-build pipelines (maybe I will regret even asking 😄).

For example, if it turns out that any SDK can be built by bootstrapping using .NET Core 1.0, maybe it will turn out to be quite a viable path. However, early versions are likely to have plenty of linuxism so whether this is a viable path needs to be seen.

Anyway, clarifying the minimal SDK version we need to target for bootstrapping will be helpful even for setting up the cross-build pipelines (in an ideal world we do not want to have to update the bootstrapping SDK).

@jkotas
Copy link
Member

jkotas commented Mar 19, 2022

,NET of given version needs .NET SDK of the same version to bootstrap, bootstrapping on anything lower is not realistically possible.

@Thefrank
Copy link

Thefrank commented Mar 19, 2022

Using Microsoft's feeds you can start at 3.1. This will be an ONLINE build however:

so 3.1 (with patches) -> 5.0 (with patches) -> 6.0 (with patches) -> HEAD/7.0 (no patching needed)

EDIT: @jkotas beat me to it

@spchamp
Copy link

spchamp commented Mar 21, 2022

I've added a new scripting interface for building with the cross-build repository all in docker. The scripting still needs some edits, however.

Before integrating it fully with a patch set for a contrib fork of the cross-build repository, I'm trying to test it out with Docker on Debian 10. Ideally, this would be pursuant to testing it in Docker under GitHub Workflows/GitHub actions.

In a partial test for the Docker image, the build was failing in the runtime repository.

/runtime/.dotnet/sdk/6.0.100-rc.1.21430.12/NuGet.RestoreEx.targets(19,5): error : '6.0.0-rtm.22171.093215' is not a valid version string.

This may have been due to a version mismatch? namely for the SDK installed under runtime/.dotnet and the SDK version specified in runtime/global.json.

External to the .NET build scripting, the top-level scripting for this build-it-all-in-docker approach would cache and install an SDK of the version in each repository's global.json under each repository's .dotnet for each repository before the build. That was skipped for the initial test here.

The build under Docker may have drawn in some artifacts from when I was trying to test the same scripting outside of Docker. Those builds outside of Docker were not successfull.

/home/me/.nuget/packages/microsoft.dotnet.arcade.sdk/6.0.0-beta.21522.3/tools/Version.BeforeCommonTargets.targets(57,5): error MSB4186: Invalid static method invocation syntax: "[MSBuild]::Add($([MSBuild]::Add($([MSBuild]::Multiply($(_BuildNumberYY), 1000)), $([MSBuild]::Multiply($(_BuildNumberMM), 50)))), $(_BuildNumberDD))". Method '[MSBuild]::Add' not found. Static method invocation should be of the form: $([FullTypeName]::Method()), e.g. $([System.IO.Path]::Combine(`a`, `b`)). Check that all parameters are defined, are of the correct type, and are specified in the right order. [/usr/local/src/crossbuild/build/runtime/Build.proj]

Reviewing that this morning, it seems that the build ID must have a specific syntax.

Referring to the original build.sh in the cross-build repository form Thefrank, I'll update the scripting for that part and retry the added build scripting outside of Docker.

@spchamp
Copy link

spchamp commented Mar 25, 2022

While testing the patch for adding a Docker-focused build scripting in the cross-build repository, I've tested the patched code on a Debian 11 bare metal install and with a chroot environment on FreeBSD, the later installed with the sysutils/debootstrap port.

The patch would be an adaptation of the original build.sh at the dotnet-freebsd-crossbuild repository. I was hoping that it could all be run within a single docker image (or independent of docker). I've been testing it outside of Docker, before testing any new Docker images for it.

In a couple of .NET runtime releases for .NET runtime and AspNetCore version tags in the 6.100 series, the build fails persistently with errors similar to the following:

/usr/local/crossbuild/build/runtime/.dotnet/sdk/6.0.101/NuGet.RestoreEx.targets(19,5): error : '6.0.3-dev.22175.024243' is not a valid version string. (Parameter 'value') [/usr/local/crossbuild/build/runtime/Build.proj

I'm not certain if a workaround can be patched on to that. Those version strings, I think, are hard-coded into the runtime repository - it might be kind of etched into the chrome, for all of the cross-repository version dependencies?

The build has presumably not failed as so, on any Windows runtime environments?

Such an error about version strings was occuring when beginning with dotnet-sdk-5.0 or dotnet-sdk-6.0 under Debian 11 on a bare metal install and Debian 11 under chroot on FreeBSD. Maybe there's something in the Microsoft docker images that somehow prevents such an error from occuring (??)

Seaching for a file presenting that version string to the build, nothing actually shows up with an exact match. There are version strings with a similar syntax, throughout the *.assets.json files under the artifacts/obj directory in the runtime build filesytem - with a version string 6.0.3-dev.22174.18380. Maybe this syntax could be related to DLL signing, in the suffix after the 6.0.3-dev part?

Maybe it could work out under some earlier release tag. The message above was from a build for the dotnet-runtime repository under the v6.0.3 release

@spchamp
Copy link

spchamp commented Apr 6, 2022

Starting with a new openSUSE Tumbleweed installation under Bhyve (EFI/GPT in the vm-bhyve configuration, accessing via ssh for build) I'm seeing some more success with the local adaptations on the cross build scripting.

Using the .NET Core runtime and AspNetCore repositories each at release tag v6.0.3, with the installer repository at tag v6.0.103, with patches from dotnet-freebsd-crossbuild for the dotnet upstream repositories, then in running the adapted build directly on openSUSE, it at least reaches the cmake part of the cross-build.

At eng/native/tryrun.cmake the build seems to expect a file ${CROSS_ROOTFS}/bin/freebsd-version pursuant to setting the FREEBSD build flag. I'm assuming that this file must exist under a filesystem on the Linux machine for the cross-build.

After some tooling with FreeBSD /usr/src/release/Makefile under a local FreeBSD 12.3 build on the stable/12 branch, I have a base.txz available. I'll try installing that under the CROSS_ROOTFS dest then running the build again.

iirc the docker scripting that has been available for the cross-build was using Ubuntu 18.04 in the docker instance. Maybe there'll be some more progress starting with openSUSE and Bhyve?

For community review, I'll add my adapted build scripting to my own contrib fork on dotnet-freebsd-crossbuild then add a back link here

For automating the build, If using GitHub for the build automation, GitHub may not have any GitHub Actions support for Bhyve as yet. They do support Docker. I'm sure there must be a Docker image available for openSUSE, as a starting point.

If automating the build with Bhyve, there's vagrant (ruby)?

Alternately, for automating the build with openSUSE for the cross build, if developing something like an intermediate distribution channel in RPM form for the cross-built parts, there's openSUSE OBS. If I can get the build worked out locally, it should be fairly straightforward to port it to an RPM spec file for the OBS build. Theoretically, this distribution approach could be applied as to produce multiple versions of the cross-built parts under one RPM repository, using openSUSE' distribution services. Those could then be used from any single port build for FreeBSD, obviating any theoretical virtualization tooling in the port build.

So, hopefully it works out past this cmake part, on openSUSE. I'll work on publishing the added cross-build scripting, then back to the build in the cmake part.

@spchamp
Copy link

spchamp commented Apr 6, 2022

An entry point for the docker-oriented cross-build scripting, but for running the cross-build on some Linux outside of docker : entrypoint_local.sh. This script calls actions/entrypoint.sh with a build environment configured for building outside of Docker. This repository is a contrib fork on dotnet-freebsd-crossbuild

Tools needed in the build environment: jq; git; patch; sed; tar; gzip; wget; dotnet; bash

Working on an update that installs a FreeBSD base.txz (--no-fflags) for a cross-root dir before the build (done), now to figure out the cmake parts in the scripting for the dotnet/runtime cross.

@Thefrank
Copy link

Thefrank commented Apr 7, 2022

Here is how the crossrootfs is generated for the docker images:
https://github.com/dotnet/arcade/blob/549523c3fc8929da1a3073d1a97f298e0d1dc342/eng/common/cross/build-rootfs.sh#L297-L313

It has been more than a year but the environment setup from that allowed for crossbuilding the FreeBSD runtime from inside Ubuntu (18.xx??)

I might have some time over the weekend to tinker with this but no promises

@arrowd
Copy link

arrowd commented Apr 7, 2022

At eng/native/tryrun.cmake the build seems to expect a file ${CROSS_ROOTFS}/bin/freebsd-version pursuant to setting the FREEBSD build flag.

This file as a script that works like uname -r.

@spchamp
Copy link

spchamp commented Apr 7, 2022

It seems that it will need some packages from ports installed under the cross-rootfs, i.e under the FreeBSD base system that will be installed temporarily under some Linux environment. For this, of course it might need a look at the FreeBSD ports tree. So far, I've discovered that these pkgs from FreeBSD will need to have been installed under the FreeBSD base system in the cross-rootfs dir:

  • libunwind (devel/libunwind)
  • icu (devel/libicu)
  • lttng-ust on FreeBSD (sysutils/lttng-ust) and probably its dependencies, including liburcu (sysutils/liburcu)
  • libinotify (devel/libinotify)

For ensuring lttng-ust would be available at least for testing, I've sent a tenative patch for sysutils/lttng-ust to FreeBSD bugzilla. This would update the port to the latest upstream release, 2.13.2. Candidly, I understand that I may not be well enough familiar with the C language and that project's source API enough to really test the patch or to provide any substantial quality assurance about it. At least the headers and library from the pkg might be available now, for local testing at least. Hopefully, the cmake part of the cross-build can now complete that part of the build.

Of course, I haven't tried building the FreeBSD pkg tools on Linux. I've used bsdtar to extract the pkg files under the crosrootfs dir. It seems that the cmake scripting was able to find the headers and library files from there.

For scripting the FreeBSD pkg files-installation for an automated build, it looks like the FreeBSD packagesite.txz file contains some JSON data. Like in the original build.sh and the added scripting in contrib, and once all of the deps in FreeBSD packages are available from some package repository, of course the JSON can probalby be parsed with jq, towards automating the pkg retrieval in the cross-build under Linux. For the testing insofar as for the cmake part of the buiild, I've been using pkg builds from my local Poudriere package tree, with manual scp and tar/bsdtar.

I've updated the scripting locally for the part, "Install a FreeBSD base system under a Linux environment". It might serve to link the build to some major/minor version pair under FreeBSD - I'm testing it with FreeBSD 12.3 locally. I'm not all sure of how to approach the pkg part for scripting. I'm sure it will use jq. I'l need to put a web server together for the pkg repository at site, though - might try Ruby on Rails for this ...

After some more cleaning in the dotnet source trees I was working with locally, then adding a 'clean' action to the entrypoint.sh script, a lot of the build errors I was seeing with the version-mismatches have cleared up.

Building under openSUSE, the dotnet/runtime cross-build has reached a part of the cmake build where it needed those pkg libraries and headers available in the cross-rootfs, before generating the makefiles or the ninja.build file.

Hopefully the patch for lttng-ust may work out. I'll try to update the scripting for fetch from some local kernel and pkgs distribution for testing the cross-build

@omajid
Copy link
Member

omajid commented Apr 7, 2022

libunwind (devel/libunwind)

FWIW, there's a bundled copy of libunwind in dotnet/runtime. Perhaps you can use that instead (at least initially)?

This would update the [lttng-ust] port to the latest upstream release, 2.13.2

Please note that 2.13 is actually incompatible with .NET runtime: dotnet/runtime#57784. It might crash or just not work.

@spchamp
Copy link

spchamp commented Apr 7, 2022

libunwind (devel/libunwind)

FWIW, there's a bundled copy of libunwind in dotnet/runtime. Perhaps you can use that instead (at least initially)?

This would update the [lttng-ust] port to the latest upstream release, 2.13.2

Please note that 2.13 is actually incompatible with .NET runtime: dotnet/runtime#57784. It might crash or just not work.

Oh these side effects. TY for mentioning.

If the build is building dotnet/runtime from source, maybe the soname linkage may not be an issue for the FreeBSD cross-build? I can only hope. Will see how it goes, hopefully, once the build-deps are worked out. Good to know if lttng-ust 2.13 may not be ABI-compatible with 2.12, I'll add a note to the port update ticket. TY!

fwiw with dotnet testing under openSUSE at least, openSUSE publishes liblttng-ust1 at 2.13.0. Reviewing the RPM dependencies for the dotnet-sdk-6.0 components installed on the openSUSE machine, liblttng is not listed there. It doesn't look like they've bundled the library in the RPM dist. Of course, the dependencies may be diferent for other SDK versions.

I've not yet used the dotnet tooling extensively under openSUSE, I was trying it out for building the omnisharp-roslyn C# language server with its cake build tool (old source code, newer is probably here). I don't know if any of the build errors I was seeing may have been due to lttng-ust. I'll try to find a clean test case for this, lol.

In the cross-rootfs fwiw it also needs a Kerberos impl (heimdal or mit krb5) for the gssapi support.

I'll try to publish an update for the build script, it may not yet include any support for selecting and installing pkgs under the cross-rootfs. The build is building locally - it's in the cmake part of the dotnet/runtime build, compiling with clang. So, Yay!

The cmake scripting for the dotnet/runtime cross build was trying to locate the libraries and header files under the FreeBSD cross-rootfs. Assuming all of the build deps can build under ports, I've been trying to install from a local ports tree at least for testing the build-scripting. Albeit, with some patches applied to the local ports tree, perhaps it's not a clean build per se. The defaults will all be configured for a FreeBSD base system and FreeBSD pkgs, assuming all the deps are available w/ FreeBSD core

Published the latest updates for the contribution in build scripting under the contrib branch for the cross-build

For the build deps, I'll try setting up a new bhyve VM using an official FreeBSD 12.3 release, then building the cross-build deps from the local ports tree under that VM, without any local make.conf additions and with only the patches for the liblttng-port under the portsdir. The resulting pkg files could then be published separately - unofficial pkg files. These might at least serve to facilitate a build for a FreeBSD 12.3 cross, with all the build-deps available. With root on ZFS under the VM, it should be easy enough to roll it back to the baseline installation, for each build. Sure, it's no Poudriere though lol.

If anyone may be interested in taking a look at the contributed build scripting, while a liblttng-ust built for FreeBSD might be required for the cross build, I'll publish a link to some project here at GitHub once the pkg dependencies would be available for the build.

fwiw the build failed at a point that may be beyond my own range of experience with programming languages.

I will try to take a look at the source code though, and maybe running this under Build EAR (Bear, FreeBSD port devel/bear) to record what the compilers are doing.

  /usr/local/src/dotsys_wk/crossbuild/build/runtime/src/coreclr/debug/daccess/daccess.cpp:7264:38: error: invalid suffix 'x' on integer constant
      *dacTableAddress = baseAddress + DAC_TABLE_RVA;
                                       ^
  /usr/local/src/dotsys_wk/crossbuild/build/runtime/artifacts/obj/coreclr/FreeBSD.x64.Release/inc/dactablerva.h:1:24: note: expanded from macro 'DAC_TABLE_RVA'
  #define DAC_TABLE_RVA 0x          6c0df8
                         ^
  1 error generated.
  [2386/2454] Building CXX object debug/daccess/CMakeFiles/daccess.dir/dacfn.cpp.o
  [2387/2454] Building CXX object debug/daccess/CMakeFiles/daccess.dir/enummem.cpp.o
  [2388/2454] Building CXX object debug/daccess/CMakeFiles/daccess.dir/dacdbiimpl.cpp.o
  ninja: build stopped: subcommand failed.
  /usr/local/src/dotsys_wk/crossbuild/build/runtime/src/coreclr

This was with a dotnet/runtime source tree at v6.0.3. Compiler is clang-13 from OpenSUSE Tumbleweed

@Thefrank
Copy link

Thefrank commented Apr 8, 2022

I noticed the #define DAC_TABLE_RVA 0x error occuring under crossrootfs builds that were not using docker. I never figured out WHY that was happening and just moved to docker because it was the easiest way to fix the issue at the time :(

Also, as a heads up for net7.0+, lttng is no longer used in the build process for FreeBSD and OSX. If you are sourcebuilding net6, you will still need it, but starting in net7 it just falls back to dummyprovider (like OSX). I would like to see support for DTrace for FreeBSD and/or OSX but I doubt that will happen.

@spchamp
Copy link

spchamp commented Apr 8, 2022

I noticed the #define DAC_TABLE_RVA 0x error occuring under crossrootfs builds that were not using docker.

fwiw in tracing it across the build files, with dotnet/runtime version v6.0.3 there is the following in dotnet/runtime source file src/coreclr/debug/daccess/CMakeLists.txt[source] This seems to provide a conditional configuration for a script at src/coreclr/pal/tools/gen-dactable-rva.sh for generating the dactable symbol file

if(CLR_CMAKE_HOST_FREEBSD OR CLR_CMAKE_HOST_NETBSD OR CLR_CMAKE_HOST_SUNOS)
  add_definitions(-DUSE_DAC_TABLE_RVA)

  set(args $<$<NOT:$<BOOL:${CLR_CMAKE_HOST_OSX}>>:--dynamic> $<TARGET_FILE:coreclr> ${GENERATED_INCLUDE_DIR}/dactablerva.h)

[sic]
endif(CLR_CMAKE_HOST_FREEBSD OR CLR_CMAKE_HOST_NETBSD OR CLR_CMAKE_HOST_SUNOS)

in gen-dactable-rva.sh [source] I think it's due to this:

value="$(printf "%016s\n" [_sic_]))

This adds whitespaace to the head of the value interpolated from input to the script, which is then produced vertabim in printf "#define DAC_TABLE_RVA 0x%s\n" "$value". I'll try updating the runtimeRTM patches for this

To pad with leading zeros and encode as hex, perhaps it could have used:

value="$(printf "%016x\n" [_sic_]))

fwiw the value is used conditionally, in the definition of GetDacTableAddress in src/coreclr/debug/daccess/daccess.cpp[source]

The dactablerva.h genrated by the tools script had contained only a single line, illustrating where the whitespace was introduced in the formatting:

#define DAC_TABLE_RVA 0x          6c0df8

imo with as well organized as the patches are for this cross-build, this update would be trivial, to simply remove the whitespace formatting spec from the initial printf in that script

Also, as a heads up for net7.0+, lttng is no longer used in the build process for FreeBSD and OSX. If you are sourcebuilding net6, you will still need it, but starting in net7 it just falls back to dummyprovider (like OSX).

Good to know. I wonder if there may be any options for provider selection in earlier releases?

fwiw I'd noticed the .NET 7 release tags in git ls-remote. If it was not for the original crossbuild repository, tbt I would not even be able to guess my way through this build, lol. (still catching up)

I've been focusing on .NET 6 mainly due to it being after (??) .NET Core and that it's referenced in at least one project I was trying to build.

If I can work out the additional build tooling for a .NET 6 build, then after some tooling for a distribution channel and for independent verification of the build, I'll try to focus on producing a .NET Core build and something in the .NET 7 sources.

Of course, there may be a few options available for automating the base-system build and the cross-build. After a successful cross-build with any single service mix, I think it should be possible to produce a port that would assemble the installer parts for a port installation on FreeBSD.

If there's a docker image for FreeBSD that could be used with the GitHub Workflow/Actions scripting, it might even be possible to produce a pkg build for the port, for installation to any system using a FreeBSD base system and kernel corresponding with that used in the cross-build.

I would like to see support for DTrace for FreeBSD and/or OSX but I doubt that will happen.

fwiw for the local build testing, in the machine that is running the Bhyve system, which is running the openSUSE system that I'm testing this with, the FreeBSD host was built with WITH_CTF=Defined in src.conf (handbook ref available). Though the kernel/base system build on this machine was produced without debugging symbols, and after the documentation in the FreeBSD Handbook, I believe that that feature may have served to enable the dtrace support in the FreeBSD machine. Albeit, it may not be possible to debug the kernel or base system userspace on that installation. I believe DTrace is available at least for builds from ports, on that machine.

If that build option may not have been used in the base system installed under the cross-rootfs, perhaps the dotnet/runtime cross-build may not produce any support for dtrace in the built objects. If it was available in the original base.txz for the cross-rootfs, maybe that option would be available under the dotnet/runtime cross build? (assuming dtrace support is available on the installation machine)

After taking a look with grep for the coreclr sources, it seems that a DTrace option may be available under some build configurations. If this can be worked into the cross-build for FreeBSD, it would probably not be difficult to add an additional feature to the distribution channel - to enable dtrace in the base system for the cross-build and then in the cross-built objects.

tbt while I've been focusing on eventually trying to script the build with GitHub actions and Docker, I'd like to be sure that this can be built without GitHub actions scrpiting. If it can be produced with bhyve, ideally the whole build could be produced with one machine. This might make it easier to add and test different build options across the cross-build distribution channel. Furthermore, it could serve to ensure that the cross-build would use the base.txz and build-deps from ports, using any local FreeBSD base system build and e.g a pkg tree built with Poudriere.

Poudriere itself might provide some support for automating a kernel build, such as in producing a builder system for port builds. Maybe it could be used in more of the automation here. Of course it might also need bhyve, if building for a target release outside of some version range for the FreeBSD release on the host machine.

For the dtrace option, if there may not be an official FreeBSD build with dtrace enabled, then an unofficial build + corresponding build-deps might be required for that feature. I think it can be worked out, with some scripting for Bhyve and poudriere.

@spchamp
Copy link

spchamp commented Apr 10, 2022

I was able to build a working dotnet SDK using openSUSE Tumbleweed in Bhyve, an official FreeBSD 12.3 release, and some local port builds, starting with the dotnet-freebsd-crossbuild tree.

For addressing the initial build failure in the dotnet/runtime section, and for addressing some issues I was seeing under the tests in the aspnetcore section, I've added a couple of trivial updates on the the original patch set from dotnet-freebsd-crossbuild. For the patch additions for the cross-build with dotnet/runtime, and for a certain node.js part under the testing for aspnetcore, I've sent the patch sets to upstream with separate pull requests.

I was using the entrypoint_local.sh script, which I've been working on in a contrib branch. This calls actions/entrypoint.sh such that could ostensibly be used for automating the build with Docker under a GitHub Workflow definition. The scripts can be used independent of Docker.

The most of the documentation is in entrypoint.sh. There are build dependencies listed there, including those for the build host and those for the cross-rootfs. The cross-rootfs itself and the build dependencies would have to be manually installed.

For the cross-rootfs, I'd used a base.txz from 12.3-RELEASE/ unpacked with bsdtar --no-fflags to the directory denoted as CROSS_ROOTFS to the build scripts. I'd then installed each build dependency from a package bult in a local port build for a 12.3 FreeBSD base system and kernel build. This crude package installation process did not offer any support for post-install scripts or retaining package metadata. It simply provided the files at some location under the cross rootfs, for the dotnet/runtime cross build.

The build dependencies are listed in the script. For the GSS-API option, I'd used an mit krb5 build from the local ports tree.

In order to build the lttng-ust build dependency for the cross-rootfs in the build, I'd made a patch on the local ports tree. The diff has been submitted to the ports section at FreeBSD bugzilla.

After some fixes for the scripting I've been tooling with, then notwithstanding the quality of the local network, but it went pretty straightforwardly. To my best estimate with the tree I'm working with locally, the build needs up to 16 GiB of filesystem space.

I'm going to take a look at running openSUSE in Docker ... lol under openSUSE in bhyve. If it may serve to provide a way to test out the entrypoint.sh script in a docker environment, hopefully it will be possible to automate the build and to put together some way to build the build dependencies independent of my local FreeBSD installation.

Assuming that there would be a suitable range of SDK versions available from the cross-build approach, I think it may be possible to start building the runtime and SDK parts natively on FreeBSD. I'll try to test out the scripting under Docker and put something together for automating a build with GitHub Workflow Actions.

I've tested out the SDK on FreeBSD for building a local VS project and for building the OmniSharp LSP support. Some network issues have prevented the later from completing in any resonable time tonight, but it looks like the cross-built SDK holds together, lol

@nkosi23
Copy link

nkosi23 commented Apr 10, 2022

@spchamp I am both excited and intrigued by your progress because there is one thing that isn't totally clear to me: what is the goal?

I tend to think that your end goal is to 1) create a FreeBSD port using bhyve behind the scenes to build the SDK, and 2) enable the .NET team to build for FreeBSD more easily upstream by providing them a docker image that they can automate using Github Actions.

If this understanding is correct, for 1) are we sure that the build system of the official FreeBSD package repository will allow us to run bhyve to build the port? There are all sort of limitations there such as the fact that it is not possible to fetch anything after the fetch stage of the build progress.

For 2) my (limited) understanding is that the .NET team uses Azure Pipelines to build .NET so I struggle to understand how the Github Action integration will help them. At the same time since docker is not supported on FreeBSD I assume that this is not intended to be used by FreeBSD users, so the goal of the Docker part of your effort is obscure to me.

Could you please help me clarify the goal that you have in mind

@spchamp
Copy link

spchamp commented Apr 11, 2022

@nkosi23 although I've been looking at a couple of ideas for how to automate the build outside of my own own VM configuration, I don't believe anything is really concrete about it as yet. Presently, I'm simply trying to test the build under a certain VM configuration using openSUSE. I was pretty glad that the build actually worked out, though.

After some further reading about some ostensible options for server-side automatation in some cloud network, I understand that it may not be possible to build the FreeBSD build dependencies under Docker here at GitHub.

With regards to any ostensible Docker-based orchestration for build automation at GitHub: Although I have an openSUSE installation under Bhyve available for some initial testing, but it seems that the GitHub Workflow documentation favors Debian-based distributions for Docker automation. I'm assuming that that's what they're running on the servers - maybe they're using an Ubuntu release of some version. So, I'm going to test the build under a Debian-based distribution locally, before putting together anything in Docker with GitHub.

That would not serve to address the concern of the missing build-dependency for lttng-ust. Without some kind of a patch for the ports tree, that port might not build for amd64. At last check, it seemed that there is not any pkg available for that port, on amd64, in official FreeBSD port builds at this time. If it may be possible to build a patched port tree for purpose of the build dependencies, of course that would have to be conducted outside of Docker - unless there would be some way to build a FreeBSD base system and a small number of ports for a FreeBSD system, within a Linux compiler environment, lol.

If automating the build-dependencies part in some FreeBSD system, perhaps it could also serve to provide any build options for different GSS-API implementations within the cross-build environment and thus within any installation environment (e.g MIT krb5, heimdal, or krb5 in the base system) and for DTrace support in the build. Coordinating this with some Linux-based system for the cross build may not be posible singularly at GitHub. I'll try to take a look at what I may be able to put together for an independent build server with Azure DevOps or alternately, DigitalOcean, for automating that part of the build. Perhaps it might be too much of a hack to try to build a FreeBSD base system and any number of ports, if within qemu with Docker orchestration at GitHub.

Assuming that the entire build process can be automated for any single set of .NET SDK and framework versions, perhaps it may be possible to automate some method of traversal across different SDK and runtime versions, for building any additional releases with a similar methodology.

Subsequently, perhaps the cross-builds produced under Linux could be used to produce any builds independent of the cross-buld orchestration, on some FreeBSD system - ostensibly independent to anything related to any FreeBSD brand name, other than in the uname aspects, and source and distribution rights.

Assuming then that this will have to be produced entirely independent of anything related to any official FreeBSD project, other than in the source code and legal aspects in the software distribution, then in order to make use of any such unofficial .NET SDK and runtime builds on a FreeBSD system - there are a few more bases to cross, before that final stage in the distribution concern.

For local builds, of course there's Bhyve, or Linux Containers, or the ostensible "bare metal" Linux installation, or even Qemu, furthermore any analogous systems that may be available on other operating systems.

Under one configuration, the buld has worked out with openSUSE under Bhyve.

Candidly, that is about the limit of what I have in mind. Health, all.

@MichaelSimons
Copy link
Member

To my best estimate with the tree I'm working with locally, the build needs up to 16 GiB of filesystem space.

@spchamp, In case you are not already using it, there is a /p:CleanWhileBuilding=true build option that will reduce the size needed to build.

@danfe
Copy link

danfe commented Apr 4, 2023

So, how is the progress? Did we come any closer to having modern .NET stack in FreeBSD ports? I've noticed that @sec is updating his repo more or less regularly (last commit being this year), are there any connections between his work and upstream?

In FreeBSD ports, we currently rely on Mono to build C# projects, and recently some developers become more feared it's being phased out, sometimes dubbed as "dead upstream". While this is not quite true AFAICT, there's clearly an increasing demand for modern .NET implementation which FreeBSD is unfortunately lacking.

Update: I've discovered a more actively maintained issue with very recent status update. Sorry if it was referenced earlier, I must have missed it in this case.

@Thefrank
Copy link

Thefrank commented Apr 4, 2023

@danfe very close. There are a few issues I noted in https://github.com/dotnet/runtime/issues/14537. There are two matching PRs open to fix the R2R issue. The others need to be opened/fixed here in source-build.

After the above PR's, the build process would look like:

download pre-built SDK
clone dotnet repo
unpack SDK into repo
./prep.sh
manually add missing NuGets (issue noted above)
manually patch FreeBSD RIDs into installer
./build.sh --online

Additional considerations when building:

  • No official support means manually downloading and unpacking a pre-built SDK (noted above)
    • There are three public sources for this: @sec has native built ones for x64 and ARM64 for FreeBSD 13 and both I and the Servarr team have crossbuilt ones for FreeBSD 12 x64
  • No official support means manually downloading the missing FreeBSD NuGets (noted above)
    • Both I and @sec provide NuGet feeds with our prebuilt items
  • No official support means manually patching RIDs into installer (noted above)
    • Intentional. FreeBSD is community supported. We do not want to create a massive support headache for Microsoft if people think there is official support so the RID must be manually patched here
  • allow.mlock=1 in jail when building/running
  • AFAIK offline building is still not supported for source-building, which means the ports system can not build this
    • This will hopefully be fixed before net8 launches

@sec
Copy link

sec commented Feb 4, 2024

@danfe we now have .net 8 port in FreeBSD ports tree, ready for use (x64 and arm64).

@Thefrank
Copy link

With lang/dotnet in FreeBSD port I think think this is safe to close?

The port uses the VMR with a prebuilt bootstrap and some script work to make sure it works under the constraints of FreeBSD's build and pkg servers.

Those wanting a portable build can always use either a) cross-building from Linux or b) native non-VMR building. The patches from ports should be enough but there are a few PRs for future versions of dotNET that portable builders should either cherry-pick or wait for them to get merged (e.g. FreeBSD RID, OpenSSL3 support)

@joperator
Copy link
Author

With lang/dotnet in FreeBSD port I think think this is safe to close?

This FreeBSD port is very good for installing a ready-to-use .NET SDK on FreeBSD (something I've wanted for years). However, maybe I'm a bit traditional, but I'd still like to have some kind of documentation on how to build the .NET SDK on or at least for FreeBSD using native build and/or cross-build on Linux.

@Thefrank
Copy link

@joperator the port uses dotnet's VMR to generate a non-portable build. if you have a ports tree setup, you can just make build in lang/dotnet and it will produce all of the items need to not only install the SDK (make install works too) but build SDK minorversion+1 from the VMR.

If you want portable native builds @sec has an excellent set of scripts setup for that and even publishes SDKs with a handful of NuGets

If you want portable cross-builds from Linux I have a repo for that. I publish the SDK along with NuGets too. I do however need to spend some time improving the scripts as the default container switched from ubuntu to cbl-mariner and that breaks a number of things.

Both of us maintain our own NuGet feeds if you just need NuGets.

My general advice is to use the port as pkg install dotnet is easier and quicker than the other options. If you want to build a portable version @sec scripts will be more useful for pure FreeBSD users. My repo will be better for those that want to produce FreeBSD items but work in Linux/OSX

Finally, for official documentation? Well, FreeBSD is still a community supported platform so it only has instructions to build the runtime repo and nothing more. The FreeBSD wiki entry might be a better place but it needs some updates. The only major difference between natively building and cross building from Linux is that the native builds need a working SDK and a handful of NuGets setup before hand while the cross build can download just about anything it needs from Microsoft's feeds. The cross build requires docker unless you want to create your own crossrootfs.

@joperator
Copy link
Author

It's not an official documentation, but your comment is something I asked for: A quick overview where to find scripts and instructions to build the .NET SDK on/for FreeBSD. Thank you very much and it's great to see how far the community has come with this. 🎉

@arrowd
Copy link

arrowd commented Feb 14, 2024

Maybe we should put some additional code to the port to ease cross-compiling? Like, some knobs or targets hidden in the Makefile that may be adjusted/ran by the end user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

No branches or pull requests