[question] Possible Workflow with CMake (and CLion)

[TorstenRobitzki]

Hello,
I'm currently working on a project with a small team working on a relative small code base. Nevertheless, the code base has some dependencies on external libraries and these external libraries have their dependencies on their own. Up to now, we use a shell script to install the dependencies in docker. The exact versions of the installed dependencies where more or less random and depended on the time of execution.

I've convinced my team to use Conan to install the dependencies and thus to codify our dependency and to pin the exact version, we are using with a specific version of our own project.

We use CMake for the build. We have 3 different build configurations (gcc-debug, gcc-release, clang-debug). So far, I was able to configure Conan to install our dependencies (using the 3 configurations above) and to build the software and their tests.

Now I have a pretty large command line for installing every configuration, which I probably could shorten a little bit, by using profiles and a large command line for the CMake configuration, which I can shorten largely by using presets.

Now, I would like to integrate Conan into CMake. I had the dream, that I could simply checkout our sources and use CMake to configure and build the software and that Conan is integrated into the CMake build, so that it gets called to determine the required include and library paths and of cause, to install missing libraries / dependencies. And of cause, if our Conan configuration changes (which simply can happen, by switching the branch), I don't want to explicitly run Conan, but use CMake to detect the required update.

The preferred method to integrate both, seems to be the CMakeToolchain integration. But in this case, the required toolchain file has to exist already before I call CMake (and as there is a chance, that the project has to be cross build in the future, we probably are going to use the tool chain feature of CMake on our own).

Is there a seamless integration of Conan into CMake that doesn't require me to call Conan before I call CMake? Is my expectation, about what Conan should be used for / should be capable of, somehow wrong? Is there an example project accessible, that integrates CMake and Conan).

best regards,
Torsten

• I've read the CONTRIBUTING guide.

[memsharded]

Hi @TorstenRobitzki

I guess that you are talking about the https://github.com/conan-io/cmake-conan integration, but you are concerned about its limitations.

That project was born by the desire of some users to run just cmake, and let it call conan automatically. Even if I didn't love this integration, it worked relatively well with the previous CMake integrations (cmake generator).

Relatively well, means that it has its issues, for example, need to deal with multi-configuration projects, installing Debug/Release configurations in one pass, further polluting every CMakeLists.txt that relies on it.

But on the other hand, we got a huge pressure to implement a "transparent" integration, that is, one that didn't need to modify anything in your CMakeLists.txt at all. That was the origin of CMakeToolchain and CMakeDeps. The pressure continued to produce CMakePresets files too. These integrations are more powerful, flexible and complete than the previous ones, so we are happy with them, and they are the ones that are moving as the only ones in 2.0.

But this introduced a chicken-and-egg problem. You can't have both, it is impossible to have a transparent integration, and also have CMake call automatically Conan, because that highly pollutes the CMakeLists.txt files. But more importantly because the integration relies on generating CMake toolchain.cmake files and CMakePresets.json files, and has CMake cannot get these files after it is launched, read them from CMakeLists.txt, this is an unsolvable problem.

So the only possibilities that we see moving forward are:

• Users get a bit used to do conan install when they clone a project with Conan dependencies. This is kind of standard in other languages, you clone the project, then do npm install, or whatever to fetch dependencies. To be honest, I am loving how the CMakePresets integration is working, yes, you need to do one or more conan install for your dependencies, but the flow is super smooth and super powerful, for example, can easily define builds for different variants of the project (like different compiler versions, different architectures, shared/static), and VS Code or CLion (next version) will load them and use them automatically.
• Use higher level automation. Let it be a script on the project, some IDE plugins (we already have for CLion and Visual Studio, but they are put on hold, because they didn't get big traction), or some git hooks
• Trying a different CMake approach, like something more built-in in the CMake app, but that sounds super challenging too.

[TorstenRobitzki]

Hi James,

thanks again for taking the time to look into my concerns!

I guess that you are talking about the https://github.com/conan-io/cmake-conan integration, but you are concerned about its limitations.

No, not necessary. While evaluating our options to integrate Conan, I've also stumbled over the project above. I would love to keep the dependencies purely in our conan.py file.

The problem is, that one not just have to invoke Conan, when one do a fresh git clone, but with every git checkout / branch, one can potentially get a changed Conan configuration, that requires Conan to run. Most of the developers in my team, are used to use IDEs (CLion) and would at least require a button to press. Documenting the Conan call (on the command line) in some kind of Wiki seems contra intuitive. Especially as we try to use Conan to put as much of the required knowledge into the build system.

In our current workflow (still work in progress) to do a build (using GCC for debug build), we issue the following commands:

conan install . --profile ./tool/conan/profile_gcc_debug --build=missing --install-folder build_gcc_debug/ --output-folder build_gcc_debug/
cmake --preset gcc_debug
cmake --build --preset gcc_debug --parallel

The last step, would be omitted, when used in an IDE. There is a lot of redundancy between the call to conan and cmake and getting something wrong here can easily lead to an hour debug "ghost problems". If we could move the conan install into the cmake call, things would be way less "complex".

But in general, I think our project does not have any special requirements. Vanilla CMake, Vanilla GCC and clang, Vanilla CLion, so I though I might have problems to understand the concepts of Conan somehow.

I really like the approach to generate CMake files, that can be used by find_packages() as this makes the integration of Conan to CMake really transparent and we hadn't to change all the CMake files. What I don't like is the (miss-)use of the CMake toolchain file, as this requires to explicitly run Conan before CMake (instead of from within) and it prohibits the use of the toolchain file for its original purpose.

In the end, conan_toolchain.cmake is (in our case) just ~70 lines of code, mainly setting library and include paths. If this file would be created by a function called from CMake and would be updates, if the conanfile.py files changes or if the used profile changes, the integration would be perfect.

Thanks a lot!

best regards,
Torsten

[stb1706]

Hi Torsten,
how about

configure_file( ${CMAKE_SOURCE_DIR}/conanfile.py ${CMAKE_BINARY_DIR}/conanfile.py COPYONLY )
execute_process( COMMAND /bin/bash -c "conan install ${CMAKE_SOURCE_DIR}" )
include( ${CMAKE_BINARY_DIR}/conanbuildinfo.cmake )

in order to reconfigure when a change in conanfile.py has happend, you need to unset some package-cache variables...
Perhaps by using

set_cmake_property( VARS VARIABLES )
foreach ( VAR ${VARS})
  if ( VAR MATCHES ".*_DIR$" AND ( NOT VAR MATCHES "CMAKE.*" ) )
    message( ${VAR} )
    unset( ${VAR} CACHE )
  endif ()
endforeach ()

best regards,
Stefan

[andioz]

Hi @memsharded,

I stumbled into this issue because I have a very similar experience as @TorstenRobitzki. We are using Conan 1.x for a while now, in combination with cmake-conan and the old-style cmake generator.

Now, while trying to migrate to the newer Conan 2.x technology I have similar concerns. For example, we are bound to old CMake version (original from Ubuntu 20.04 distribution) by our customer. AFAIK this version has no preset support. I'm not sure how this will be handled by the new CMakeDeps generator?

Users get a bit used to do conan install when they clone a project with Conan dependencies. This is kind of standard in other languages, you clone the project, then do npm install, or whatever to fetch dependencies.

I cannot agree to this statement. Well yes, for example with Python virtual environments, we prepare the venv before using it. But in case of Conan, there are files installed into the build folder of the projects, not only general files. And running only CMake inside CLion is a requirement.

And finally, I'm currently really confused about the "need" to use generated toolchain files and/or preset files. We have our own toolchain files, and when producing a package including these files into the generated toolchain file at the top of the file doesn't work well.

Currently I'm collecting information about the new features, and it seems the documentation is not complete yet. But overall, Conan is a great tool, I don't want to miss it!

BR Andi

[memsharded]

Hi @andioz,

Now that we have launched 2.0, we have resumed our efforts in cmake-conan, we are experimenting with the new dependency providers, please track the efforts in conan-io/cmake-conan#473

Now, while trying to migrate to the newer Conan 2.x technology I have similar concerns. For example, we are bound to old CMake version (original from Ubuntu 20.04 distribution) by our customer. AFAIK this version has no preset support. I'm not sure how this will be handled by the new CMakeDeps generator?

CMakeDeps and CMakeToolchain can both work with CMake >=3.15 (per conan-io/tribe#4), it is not mandatory to use CMakePresets, it can be convenient, but not mandatory.

For the cmake-conan integration, it is not that clear, we will try to have some alternative for older CMake versions, but some features like the dependency providers need modern CMake versions (3.24), so we will see how it goes.

And finally, I'm currently really confused about the "need" to use generated toolchain files and/or preset files. We have our own toolchain files, and when producing a package including these files into the generated toolchain file at the top of the file doesn't work well.

There is also no "need" to use it. It can be super convenient, because it can help a lot to map the Conan settings to the CMake build, but if you can guarantee that your build is consistent with the settings, then it is good to not use it. Conan also provide ways to fully replace the conan_toolchain.cmake with a user one, or to have the conan_toolchain.cmake to include() a user one to have the best of both worlds. See:

conan config list | grep cmake
tools.cmake.cmaketoolchain:generator: User defined CMake generator to use instead of default
tools.cmake.cmaketoolchain:system_name: Define CMAKE_SYSTEM_NAME in CMakeToolchain
tools.cmake.cmaketoolchain:system_processor: Define CMAKE_SYSTEM_PROCESSOR in CMakeToolchain
tools.cmake.cmaketoolchain:system_version: Define CMAKE_SYSTEM_VERSION in CMakeToolchain
tools.cmake.cmaketoolchain:toolchain_file: Use other existing file rather than conan_toolchain.cmake one
tools.cmake.cmaketoolchain:toolset_arch: Toolset architecture to be used as part of CMAKE_GENERATOR_TOOLSET in CMakeToolchain
tools.cmake.cmaketoolchain:user_toolchain: Inject existing user toolchains at the beginning of conan_toolchain.cmake

In especial user_toolchain and toolchain_file

[andioz]

Hi @memsharded

Thank you for taking time for an answer.

Now that we have launched 2.0, we have resumed our efforts in cmake-conan, we are experimenting with the new dependency providers, please track the efforts in conan-io/cmake-conan#473

That's good news! Would really appreciate a clean solution, at best integrated in CMake or Conan itself ;)

CMakeDeps and CMakeToolchain can both work with CMake >=3.15 (per conan-io/tribe#4), it is not mandatory to use CMakePresets, it can be convenient, but not mandatory.

Well, Focal version is 3.16, that should be fine. But presets were introduced with 3.19, thats why we're not using it currently. I have not tried how conan would work with this old version, I will try when having some spare time, maybe.

There is also no "need" to use it. It can be super convenient, because it can help a lot to map the Conan settings to the CMake build, but if you can guarantee that your build is consistent with the settings, then it is good to not use it. Conan also provide ways to fully replace the conan_toolchain.cmake with a user one, or to have the conan_toolchain.cmake to include() a user one to have the best of both worlds.

Yes, I found these conf settings. Currently I'm hacking this setting into self.conf inside the generate() method, which is not really the correct way to do it, right? I tried to set it in the profile, but then I had troubles with dependencies I think, and the profile is not the right place, it's a need for one recipe, not the whole system, in my opinion.

What I currently want: having a CMake project for a library with (optional) dependencies (gtest, benchmark, ...) via conan. We are building packages with CMake and additionally a conan package with a producer recipe. Consumers will use it mostly using CMake again but we are open for other client toolchains. For me right now, the producer/consumer sides looks a little bit entangled, at least the CMake components. I liked the old cmake generator with a lightweight way to use the packages without need for find_package(). The need to use find_package() add more conditional code to the global CMakeLists.txt (well, I can put it into every subdir, but than I have duplications).

And finally, I had trouble to include "build_type=Release" dependencies within a "Debug" project - I use the "&:build_type=Debug" setting with conan install, no idea if this is the best way to map release packages to my current build type?

BR Andi

[memsharded]

Yes, I found these conf settings. Currently I'm hacking this setting into self.conf inside the generate() method, which is not really the correct way to do it, right? I tried to set it in the profile, but then I had troubles with dependencies I think, and the profile is not the right place, it's a need for one recipe, not the whole system, in my opinion.

Yes, not the best, the profile info (settings, options, conf) is read-only in recipes. While python might allow assigning values to some in recipes, that might stop working anytime, raise exceptions or have other undefined behavior. Providing it in the profiles is the very recommended way, because it is the only approach that can provide flexibility enough, hardcoding things in recipes leads to worse maintainability.

I tried to set it in the profile, but then I had troubles with dependencies I think, and the profile is not the right place, it's a need for one recipe, not the whole system, in my opinion.

That is unexpected, in general it is not possible to use a toolchain in one package that links with packages build with another toolchain without linking (or other worse, more subtle runtime) errors.
For very special cases, the per-package conf and settings is possible like: mypkg*:tools.cmake.cmaketoolchain:xxxx=yyy will only apply to that mypkg* pattern. Same for settings is also possible. For more advanced cases, it is possible to use jinja2 templating in the profiles. But this is exactly the idea, that the configuration of the dependency graph is captured in profiles, and you can see there "hey, these packages are built with this toolchain, but the other packages do not use that toolchain" as opposed of that important configuration being buried in recipes.

I liked the old cmake generator with a lightweight way to use the packages without need for find_package(). The need to use find_package() add more conditional code to the global CMakeLists.txt (well, I can put it into every subdir, but than I have duplications).

yeah, I liked it too! It was the first generator we created and worked quite well. But the push of a large majority of users was towards having a "transparent" cmake integration that was not intrusive in the CMakeLists.txt, and that is achieved with the modern integrations. We need to converge to one integration, because both the maintenance burden and also a lot of confusion for many users is not sustainable...

And finally, I had trouble to include "build_type=Release" dependencies within a "Debug" project - I use the "&:build_type=Debug" setting with conan install, no idea if this is the best way to map release packages to my current build type?

Yes, it is possible to use the CMAKE_MAP thing for this too, but it seems that properly telling Conan: "my consumer is Debug, but the dependencies are Release" works fine without needing to add extra things to CMakeLists.txt

[andioz]

OK, conf is read-only for the whole recipe, good to know. I have to retry with the profile, not sure what's the problem. But I'm still not convinced that the profile would be the best place for such details; information is spreaded over several places then, and usually we use profiles for more general settings. jinja2 templating in profile I tried but I think it didn't work (thought then its only for global.conf). But have to retry too.

cmake generator: I see, find_package() is more popular. But I hope later is time for lightweight alternatives.

So, regarding Debug/Release, using my approach looks like being the best solution.

I really appreciate the discussion culture here in Conan projects (CCI too), no question is too stupid and everybody takes time to write a proper answer. Thanks for that!

BR Andi

UPDATE: I'm on Conan 1.59 right now but want to prepare for moving to 2.x later on