[question] VirtualBuildEnv in packages that were not just built

[hlewin]

What is your question?

Hello!
At least when using conan 1 VirtualBuildEnv(self) does not seem to contain definitions from tool(or build) dependencies if the package was not built in the conan invocation.
The use case here is that we are doing some stuff in the deploy() method like setting up RPATHs, signing binaries etc. that would need dependencies given in the build-profile as tool requirements. But when the package to be deployed was not built in the same conan invocation the tool requirements do not show up in self.dependencies and hence they do not propagate any settings in VirtualBuildEnv. The only things we get there are variables from the [buildenv] section of the profile.
Is this intended behaviour and is it still present in conan2? I get it is an optimization to not instantiate build-context dependencies if not necessary but the behviour right now seems strange... - like a bug to be honest.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @hlewin

Thanks for your question.

This would be mostly intended behavior.
The VirtualBuildEnv and tool-requires are a "build-time" thing, while the deploy is a pure runtime thing, independent of the build process.

I think that in Conan 2 you might have some new functionality, for example the deployers that you can use externally to the recipes, and there is a new tools.graph:skip_binaries conf that forces to not skip binaries of dependencies, even when they are not necessary, but I am not sure if this would work for your use case.

For deployment and runtime things, we have seen users doing a 2 step approach, first conan install --tool-requires=.. (or a conanfile with tool-requires) then activating the generated environment, then calling the deploy, that will use those tool-requires because the environment has been activated in the shell.

[hlewin]

Hi @hlewin

Thanks for your question.

This would be mostly intended behavior. The VirtualBuildEnv and tool-requires are a "build-time" thing, while the deploy is a pure runtime thing, independent of the build process.

Yes and yes - which in my understanding means explicitly using VirtualBuildEnv(self) should reflect the build environment. It makes no sense that dependencies that are scoped with self.require(...., build=true) do not propagate definitions of their buildenv_info to VirtualBuildEnv. Or to put it another way: The current state of affairs is that the contents of VirtualBuildEnv are virtually unforseeable: They are defined by whatever accidentally happened to be being built in the invocation of conan.

I think that in Conan 2 you might have some new functionality, for example the deployers that you can use externally to the recipes, and there is a new tools.graph:skip_binaries conf that forces to not skip binaries of dependencies, even when they are not necessary, but I am not sure if this would work for your use case.

This is something I'll have a look at. Deciding the course of action for a transition to conan 2 is one reason of this question.

For deployment and runtime things, we have seen users doing a 2 step approach, first conan install --tool-requires=.. (or a conanfile with tool-requires) then activating the generated environment, then calling the deploy, that will use those tool-requires because the environment has been activated in the shell.

This would be no real option for the use case at hand as the question which packages and definitions are needed for a given deployment are nontrivial, ie one would have to mirror the logic of picking options and such that is done in conanfile.pys in shell scripts. - this is not feasible. The possible workaround is a "forwarder" package with no content of it's own that self.require()s the package to be actually deployed and make it being built on the command line, as this will instantiate the full build-environment. But this is not about workarounds. It is about semantics. The documentation makes no indication that a tool-requirement will not propagate it's buildenv_info to VirtualBuildEnv anywhere.

[memsharded]

The possible workaround is a "forwarder" package with no content

Sounds it could be possible.

It would be good to understand a bit more the full picture:

• Are all packages deployed?
• Do all packages contain a deploy() method?
• How many packages are actually deployed this way?
• How many different tool_requires are needed to enable the deploy functionality? What are they doing?
• Are the tool_requires defined in recipes conanfiles, or injected via [tool_requires] in profiles?
• Is it enough if they inject themselves in the PATH for finding the executables, or anything else?

The documentation makes no indication that a tool-requirement will not propagate it's buildenv_info to VirtualBuildEnv anywhere.

the semantics are that tool_requires can propagate their environment to the packages that directly tool_require it, exclusively at build time (that means build() and package()), but not outside of the build time. It might be possible to add some clarifications in the docs about this, but at least this has been the spirit and intention since Conan 1 and also still in Conan 2.

[hlewin]

The possible workaround is a "forwarder" package with no content

Sounds it could be possible.

It would be good to understand a bit more the full picture:

• Are all packages deployed?
• Do all packages contain a deploy() method?
• How many packages are actually deployed this way?
• How many different tool_requires are needed to enable the deploy functionality? What are they doing?
• Are the tool_requires defined in recipes conanfiles, or injected via [tool_requires] in profiles?
• Is it enough if they inject themselves in the PATH for finding the executables, or anything else?

We deploy a package and it's (runtime) dependencies including transitive ones. Although most of those packages implement a (actually "the same") deploy method only the deploy method of the package directly deployed matters here.
In this deploy method we need (at least) the environment variables from the [tool_requires] given in the host-profile as well as the variables given in the [buildenv] section. (Note that this is not really sound: It might be that a tool-requirement in that profile would need to be build).
Typically the tool-requirements in the profile set up their PATHs; but also setting build-system style environment variables like CHRPATH=armv7-debian-linux-gnu-chrpath or INSTALL_NAME_TOOL=llvm-install-name-tool is common.
The deploy() method of the package directly deployed makes use of such definitions to process all deployed packages (given this is functionality that should be moved to a separate generator/deployer).
We (in particular) are not so much concerned about the tool-requirements defined in the packages themselves - for those use-cases the profiles are much more important.

The documentation makes no indication that a tool-requirement will not propagate it's buildenv_info to VirtualBuildEnv anywhere.

the semantics are that tool_requires can propagate their environment to the packages that directly tool_require it, exclusively at build time (that means build() and package()), but not outside of the build time. It might be possible to add some clarifications in the docs about this, but at least this has been the spirit and intention since Conan 1 and also still in Conan 2.

Yes, clarifications would indeed be helpful in this case. Currently there are no mentions of VirtualBuildEnv being restricted to certain methods. Another (and in our use-case more helpful) possibility would be to expose a method that actually instantiates the build-environment if this hasn't already be done so.
Alternatively a way to instantiate a given environment defined in one or more given profiles would suffice for our (concrete) use-case but would still leave VirtualBuildEnv(self) somewhat dubious without clarifications.

[memsharded]

That makes sense thanks for the clarification.

I am going to check a little bit and experiment to see if there is some possible solution in Conan 2.

Just to make sure, I understand that this doesn't work either in Conan 1, and you are mostly interested in knowing if Conan 2 has some new functionality that could help with this, is this correct? Or is it something that you managed to get it working somehow in Conan 1, and now Conan 2 is more strict in that regard and not working?

[hlewin]

Just to make sure, I understand that this doesn't work either in Conan 1, and you are mostly interested in knowing if Conan 2 has some new functionality that could help with this, is this correct? Or is it something that you managed to get it working somehow in Conan 1, and now Conan 2 is more strict in that regard and not working?

No, I did not really get this working in conan 1. We are - for now - forcing rebuilds of packages as a workaround. I am making our stuff conan 2 compatible piece by piece and was fiddling around with VirtualBuildEnv and self.buildenv_info as a colleague happened to run into this problem (of missing settings). The whole thing was unnoticed for quite some time as - by coincidence - the package to be deployed happened to be freshly built in virtually all cases for other reasons.
So this caught us by surprise and the question popped up if this was still the case with conan 2: we will have to finish up the porting process one way or the other and this could have been a good reason to hurry up a little.

If there is way to get rid of such workarounds in conan 2 this would be nice. Most of the porting to version 2 did not make any structural changes to established processes. I am just evaluating options...

[memsharded]

I have made this test pass in Conan 2.

def test_deploy_method_tool_requires():
    c = TestClient()
    tool = textwrap.dedent(r"""
        import os
        from conan import ConanFile
        from conan.tools.files import save, chdir
        class Pkg(ConanFile):
            name = "tool"
            version = "0.1"
            type = "application"

            def package(self):
               with chdir(self, self.package_folder):
                   echo = "@echo off\necho MYTOOL RUNNING!!"
                   save(self, "mytool.bat", echo)
                   save(self, "mytool.sh", echo)
                   os.chmod("mytool.sh", 0o777)

            def package_info(self):
               self.buildenv_info.prepend_path("PATH", self.package_folder)
        """)
    conanfile = textwrap.dedent("""
        from conan import ConanFile
        from conan.tools.env import VirtualBuildEnv
        class Pkg(ConanFile):
            name = "pkg"
            version = "0.1"
            tool_requires = "tool/0.1"
            settings = "os"

            def deploy(self):
                venv = VirtualBuildEnv(self)
                with venv.vars().apply():
                    ext = "bat" if self.settings.os == "Windows" else "sh"
                    self.run(f"mytool.{ext}")
        """)
    c.save({"tool/conanfile.py": tool,
            "pkg/conanfile.py": conanfile})

    c.run("create tool")
    c.run("create pkg")

    c.run("install --requires=pkg/0.1 -c:b tools.graph:skip_binaries=False --deployer-package=*")
    assert "MYTOOL RUNNING!!" in c.out

the keys are:

• Explicit usage of VirtualBuildEnv with with venv.vars().apply(): in the deploy() method
• Forcing the retrieval (not-skipping) the tool-requires with -c:b tools.graph:skip_binaries=False. Note --conf:build to not skip tool-requires

I think a similar approach would be feasible using the Conan 2 new external --deployer, as they work mostly the same as the in-recipe deploy() method.

Also the approach of using a dedicated recipe that uses standard generate() + build() method to do what you would do in the deploy() method of the direct recipe could work well too.

[hlewin]

This does indeed look exactly like the thing I need - thank you for your efforts!

I did not think about generating separate package that does the "bundling" in this scenario - although we already have a similar package in place to add implicit dependencies like compiler-specific runtime libs ( libc++ and such ) from the Conan cache to a deployment. That could indeed be extended easily to do a "dumb" deployment without gathering any implicit dependencies.
All in all I think we should do the grunt-work and go through the rest of packages that need porting to Conan v2 and finish up the transition.

That having said I would still welcome a remark in the documentation that VirtualBuildEnv might not contain settings from tool-requirements that were skipped because they weren't needed for the (conan-)command to be executed. It is always better to have such things nailed down.

Again - thank you for this discussion and your efforts!

[memsharded]

That having said I would still welcome a remark in the documentation that VirtualBuildEnv might not contain settings from tool-requirements that were skipped because they weren't needed for the (conan-)command to be executed. It is always better to have such things nailed down.

Sounds good, I'll move this ticket to the docs repo.

Again - thank you for this discussion and your efforts!

Thanks to you for your feedback!