Intergrate Conan into build pipeline #1980

hobyst · 2022-07-28T22:21:29Z

Description

This PR creates a Conan recipe (conanfile.py) to allow developers to satisfy the pre-built dependency requirements without having to use pre-built binaries provided by the project. It also creates a new CMake script CopySharedLib.cmake which can be run at build time by the underlying build system to copy shared libraries and other artifacts needed in the binary folder that also follows and replicates symbolic link chains.

Requires

Fix CMake issue due to illegal target property set psvpfstools#9

To-do

Test on Windows
Need help to write a macOS Conan profile and test if everything works properly (thanks to @KhoraLee for the help)
~~Set up Vita3K/ext-deps so that people can easily obtain pre-built packages of all the dependencies needed without needing to use Conan~~

sunho · 2022-07-30T16:24:03Z

Nice, why not ready for review?

hobyst · 2022-07-30T19:03:43Z

Nice, why not ready for review?

Because I'm making some changes locally so that it can be enforced to all code related to the emulator more easily, including Boost and FFmpeg.

sunho · 2022-07-31T22:30:41Z

cmake/conan/profiles/linux-clang_libc++

+arch=x86_64
+arch_build=x86_64
+compiler=clang
+# Using Clang 11 so that no issues arise in GitHub CI, but any version newer than that can be used just fine


Do you mean clang lower than 11 fails in CI?

I mean that because the default Clang version in CI is Clang 11, setting it to a higher value would probably cause issues with CI and therefore fail the build due to the Conan preset requiring a higher Clang version than what CI actually uses.

Yup that make sense.

EXtremeExploit · 2022-07-31T22:32:24Z

i think pr name sould be changed, it turned into making linux use libc++ into making vita3k use dependencies from conan and improving their integration

sunho · 2022-07-31T22:36:55Z

Maybe worth splitting off pr if you want. I like introducing conan btw -- much better than our fragile DIY boost building.

hobyst · 2022-07-31T22:50:20Z

Maybe worth splitting off pr if you want. I like introducing conan btw -- much better than our fragile DIY boost building.

I initially just wanted to enable libc++ (Clang) for Linux builds, in which I succeeded to do so without Conan (@EXtremeExploit has been trying it as well using the first commit I made to this PR and we can confirm substantial performance improvements just by using it instead of libstdc++). But since Boost and FFmpeg were still built with libstdc++ (GNU) I needed to rework the way those two are built so that I could enforce the C++ library I wanted. I could split it into separate PRs but since I re-adapted the libc++ settings to have Conan in mind, it's more straightforward to keep both on the same PR.

sunho · 2022-07-31T23:33:56Z

Maybe worth splitting off pr if you want. I like introducing conan btw -- much better than our fragile DIY boost building.

I initially just wanted to enable libc++ (Clang) for Linux builds, in which I succeeded to do so without Conan (@EXtremeExploit has been trying it as well using the first commit I made to this PR and we can confirm substantial performance improvements just by using it instead of libstdc++). But since Boost and FFmpeg were still built with libstdc++ (GNU) I needed to rework the way those two are built so that I could enforce the C++ library I wanted. I could split it into separate PRs but since I re-adapted the libc++ settings to have Conan in mind, it's more straightforward to keep both on the same PR.

No worries. Just do in the way you prefer to. I'm totally fine with either. I just said that cause I thought libc++ needed more works to make this WIP.

EXtremeExploit · 2022-08-01T13:00:06Z

If libc++ and lld packages are now required would be nice to add them to the building instructions, same goes for conan if necessary

hobyst · 2022-08-01T19:18:47Z

If libc++ and lld packages are now required would be nice to add them to the building instructions, same goes for conan if necessary

Already did it for Conan but not for libc++ and lld. Thanks for reminding me.

sunho · 2022-08-01T23:48:56Z

If libc++ and lld packages are now required would be nice to add them to the building instructions, same goes for conan if necessary

I don't think we should default to clang/lld/libc++ family yet. It's nice to have as an option but default should be gcc/ld/libstdc++.

Macdu · 2023-03-30T10:36:56Z

From what I understand, Conan is only used for boost right now, right?

- Update GitHub action ilammy/msvc-dev-cmd and force using the Visual Studio x64 Native Tools Command Prompt - Change the depth of the Git fetch only to include the latest commit to speed up the checkout process

Make all CMake presets from the project detect the compiler launcher to use from the parent environment variables.

- Enable the project to use Boost from the Conan package manager. - Remove Boost submodule in `external/` - Prepare Conan and CMake for future implementation of FFmpeg.

hobyst · 2023-03-30T12:37:48Z

From what I understand, Conan is only used for boost right now, right?

@Macdu Yes. The FFmpeg recipe hasn't been ported to Conan 2.0 yet.

Macdu · 2023-03-30T12:56:05Z

Is openssl for windows supported ? It would be far better than using prebuilts from some 3rd party website. GitHub action is using vcpkg for openssl on Windows right now.

hobyst · 2023-03-30T16:49:34Z

@Macdu @EXtremeExploit The recipe of OpenSSL 3.x seems to now work with Conan 2.0. These are the options exposed by the package recipe configure the build:

List of options

options = {
    "shared": [True, False],
    "fPIC": [True, False],
    "enable_weak_ssl_ciphers": [True, False],
    "386": [True, False],
    "capieng_dialog": [True, False],
    "enable_capieng": [True, False],
    "no_aria": [True, False],
    "no_asm": [True, False],
    "no_async": [True, False],
    "no_blake2": [True, False],
    "no_bf": [True, False],
    "no_camellia": [True, False],
    "no_chacha": [True, False],
    "no_cms": [True, False],
    "no_comp": [True, False],
    "no_ct": [True, False],
    "no_cast": [True, False],
    "no_deprecated": [True, False],
    "no_des": [True, False],
    "no_dgram": [True, False],
    "no_dh": [True, False],
    "no_dsa": [True, False],
    "no_dso": [True, False],
    "no_ec": [True, False],
    "no_ecdh": [True, False],
    "no_ecdsa": [True, False],
    "no_engine": [True, False],
    "no_filenames": [True, False],
    "no_fips": [True, False],
    "no_gost": [True, False],
    "no_idea": [True, False],
    "no_legacy": [True, False],
    "no_md2": [True, False],
    "no_md4": [True, False],
    "no_mdc2": [True, False],
    "no_module": [True, False],
    "no_ocsp": [True, False],
    "no_pinshared": [True, False],
    "no_rc2": [True, False],
    "no_rc4": [True, False],
    "no_rc5": [True, False],
    "no_rfc3779": [True, False],
    "no_rmd160": [True, False],
    "no_sm2": [True, False],
    "no_sm3": [True, False],
    "no_sm4": [True, False],
    "no_srp": [True, False],
    "no_srtp": [True, False],
    "no_sse2": [True, False],
    "no_ssl": [True, False],
    "no_stdio": [True, False],
    "no_seed": [True, False],
    "no_sock": [True, False],
    "no_ssl3": [True, False],
    "no_threads": [True, False],
    "no_tls1": [True, False],
    "no_ts": [True, False],
    "no_whirlpool": [True, False],
    "no_zlib": [True, False],
    "openssldir": [None, "ANY"],
}

The default configuration is to compile OpenSSL as a static library with position-independent code enabled and MD2 support disabled, with everything else set as False.

default_options = {key: False for key in options.keys()}
default_options["fPIC"] = True
default_options["no_md2"] = True

Which options should be overwritten (if any) for Vita3K aside from building it as a shared library? The more things we disable, the faster it will compile.

EXtremeExploit · 2023-03-30T20:32:09Z

ig that having mostly all enabled for now is fine, we can experiment on exactly what we need from openssl as we go on imo

hobyst · 2023-04-01T13:10:21Z

@Macdu @EXtremeExploit After some more testing, it seems the Conan recipe for OpenSSL 3.x is still broken because the header files aren't included in the include path by CMake.

Even though this means that the PR should be ready to merge, I still have to make some changes before merging to re-implement download of pre-built packages because Zangetsu asked so in order to be able to build the project without using Conan and without any user interaction.

github-advanced-security · 2023-10-27T19:44:24Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

github-advanced-security · 2023-10-27T19:44:24Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

github-advanced-security · 2023-10-27T19:44:24Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

github-advanced-security · 2023-10-27T19:44:24Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

github-advanced-security · 2023-10-27T19:44:24Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

github-advanced-security · 2023-10-27T19:44:24Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

github-advanced-security · 2023-10-27T19:44:24Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

github-advanced-security · 2023-10-27T19:44:24Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

hobyst force-pushed the cmake branch 4 times, most recently from 88e36d4 to ec31566 Compare July 31, 2022 22:17

sunho reviewed Jul 31, 2022

View reviewed changes

hobyst force-pushed the cmake branch from ec31566 to 1304e03 Compare July 31, 2022 22:32

hobyst changed the title ~~[WIP] Build: Make Clang Linux preset use lld and libc++~~ [WIP] Use Conan to build Boost and FFmpeg and use libc++ in Linux Jul 31, 2022

hobyst changed the title ~~[WIP] Use Conan to build Boost and FFmpeg and use libc++ in Linux~~ [WIP] Intergrate Conan into build pipeline and use libc++ in Linux Jul 31, 2022

hobyst force-pushed the cmake branch 9 times, most recently from dc845b6 to 057f669 Compare August 1, 2022 12:49

hobyst force-pushed the cmake branch 4 times, most recently from 066ea62 to 920dae7 Compare August 1, 2022 22:13

hobyst force-pushed the cmake branch 6 times, most recently from f701063 to 1fd9738 Compare March 12, 2023 12:46

hobyst changed the title ~~[WIP] Intergrate Conan into build pipeline~~ Intergrate Conan into build pipeline Mar 12, 2023

hobyst marked this pull request as ready for review March 12, 2023 19:15

hobyst requested a review from sunho March 12, 2023 19:18

hobyst force-pushed the cmake branch 3 times, most recently from fcc5939 to 0b7a071 Compare March 15, 2023 10:00

hobyst added 3 commits March 30, 2023 13:44

CI: Force amd64 VS env and speed up checkout

523fa0e

- Update GitHub action ilammy/msvc-dev-cmd and force using the Visual Studio x64 Native Tools Command Prompt - Change the depth of the Git fetch only to include the latest commit to speed up the checkout process

Build: Make all CMake presets detect compiler launcher

5fd472f

Make all CMake presets from the project detect the compiler launcher to use from the parent environment variables.

Build: Create Conan recipe for dependencies

450681a

- Enable the project to use Boost from the Conan package manager. - Remove Boost submodule in `external/` - Prepare Conan and CMake for future implementation of FFmpeg.

hobyst force-pushed the cmake branch from 0b7a071 to 450681a Compare March 30, 2023 11:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intergrate Conan into build pipeline #1980

Intergrate Conan into build pipeline #1980

hobyst commented Jul 28, 2022 •

edited

sunho commented Jul 30, 2022

hobyst commented Jul 30, 2022

sunho Jul 31, 2022

hobyst Jul 31, 2022 •

edited

sunho Jul 31, 2022

EXtremeExploit commented Jul 31, 2022

sunho commented Jul 31, 2022

hobyst commented Jul 31, 2022

sunho commented Jul 31, 2022

EXtremeExploit commented Aug 1, 2022

hobyst commented Aug 1, 2022 •

edited

sunho commented Aug 1, 2022

Macdu commented Mar 30, 2023

hobyst commented Mar 30, 2023

Macdu commented Mar 30, 2023

hobyst commented Mar 30, 2023 •

edited

EXtremeExploit commented Mar 30, 2023

hobyst commented Apr 1, 2023 •

edited

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

Intergrate Conan into build pipeline #1980

Are you sure you want to change the base?

Intergrate Conan into build pipeline #1980

Conversation

hobyst commented Jul 28, 2022 • edited

Description

Requires

To-do

sunho commented Jul 30, 2022

hobyst commented Jul 30, 2022

sunho Jul 31, 2022

Choose a reason for hiding this comment

hobyst Jul 31, 2022 • edited

Choose a reason for hiding this comment

sunho Jul 31, 2022

Choose a reason for hiding this comment

EXtremeExploit commented Jul 31, 2022

sunho commented Jul 31, 2022

hobyst commented Jul 31, 2022

sunho commented Jul 31, 2022

EXtremeExploit commented Aug 1, 2022

hobyst commented Aug 1, 2022 • edited

sunho commented Aug 1, 2022

Macdu commented Mar 30, 2023

hobyst commented Mar 30, 2023

Macdu commented Mar 30, 2023

hobyst commented Mar 30, 2023 • edited

EXtremeExploit commented Mar 30, 2023

hobyst commented Apr 1, 2023 • edited

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

github-advanced-security bot commented Oct 27, 2023

hobyst commented Jul 28, 2022 •

edited

hobyst Jul 31, 2022 •

edited

hobyst commented Aug 1, 2022 •

edited

hobyst commented Mar 30, 2023 •

edited

hobyst commented Apr 1, 2023 •

edited