Get duplicate file names when copying dll and lib files

[leinadme]

What is your question?

Hi,

I'm using conan 2.0 on Windows 10. I'm trying to copy a bunch of dependencies such as boost, Qt, etc libs and dlls into my local build/lib and build/bin respectively. I'm using the code snippet below. It seems to work but at the end I get the following error message which is annoying. I'm not sure why I'm getting this error, the copies seem to work fine in their respective folders. Any help would be appreciated!

Thanks!

copy(self, "*.lib", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.source_folder,"build/lib"))
SameFileError: 'lib\boost_atomic.lib' and 'C:\MainFolder\applicationFolder\build\lib\boost_atomic.lib' are the same file

This is the code I'm using:

def generate(self):
    for dep in self.dependencies.values():
        if self.settings.compiler == "apple-clang":
             copy(self, "*.dylib", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.source_folder, "build/lib"), keep_path=False)
        elif self.settings.compiler == "gcc":
             copy(self, "*.so", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.source_folder, "build/lib"), keep_path=False)
        elif self.settings.compiler == "msvc":
             copy(self, "*.lib", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.source_folder,"build/lib"), keep_path=False)
             copy(self, "*.dll", src=dep.cpp_info.bindirs[0], dst=os.path.join(self.source_folder,"build/bin"), keep_path=False)

[memsharded]

Hi @leinadme

Thanks for your question
Quick question : are you trying to "re-package"? That is, create a new package from the binaries from other packages? If that is the case, I would like you to subscribe to ticket #13171, because this is something that we will consider in the 2.X roadmap.

Regarding your issue, it would be important to know if you have boost both as requires and tool_requires. Does it happens only for boost? Because as you are iterating self.dependencies it will also iterate the tool-requires, if you are re-packaging maybe this is not what you want, and you want to iterate only self.dependencies.host?

[leinadme]

I'm trying to place the files in these folders so I can compile the main application which requires these libraries. Also, for Windows I copy some of these dll's to the executable folder to satisfy missing dlls when I try to run it. The folders are temporary and then deleted after the application is built and the few dlls the app needs is copied to the executable folder. In Conan 1.x I used 'def imports' and it all worked fine.

[leinadme]

I followed the info from this link:
https://stackoverflow.com/questions/75833311/how-do-i-replace-imports-when-migrating-from-a-conan-1-x-conanfile-txt-to-a-cona

I'm also using requires but now trying tool_requires to see if that makes a difference. Seems like everything is rebuilding.

[memsharded]

I'm trying to place the files in these folders so I can compile the main application which requires these libraries. Also, for Windows I copy some of these dll's to the executable folder to satisfy missing dlls when I try to run it.

Thanks for the feedback. The recommended approach for this use case wouldn't be to copy (the copy() is the "exact" translation, but it doesn't mean there are other ways that would be better), but instead using the VirtualBuildEnv and VirtualRunEnv. With this approach:

• Recipes are typically simpler, no need to add the code to do the copies
• Recipes are faster, the copy is typically much much slower
• It is less likely to do mistakes, like not cleaning up and resulting in packaging a copy of the dependencies binaries in the current package

If you still want to go this way, we would need some way to reproduce this. Are you able to reproduce the problem by stripping out the recipe, to just requires = "boost..." and the generate() method, without the other dependencies? If there is some full example that we can reproduce, that would help a lot.

[leinadme]

The entire recipe is quite small, I get error at the 'def generate' but the files do get copied so I'm wondering why?

from conan.tools.files import copy
from conan.tools.cmake import CMakeDeps
import os

class app(ConanFile):
    name = "my-app"
    version = "1.5.4"
    license = "something here"
    author = "something here"
    url = "something here"
    description = "My Project"
    topics = ("project", "MyProject")
    settings = "os", "compiler", "build_type", "arch"
    requires = "boost/1.80.0","qt/5.15.7","qwt/6.2.0","sqliteinterface/1.1.0","sqlite3/3.39.4","dbversion/1.0.1","qt_warning_macros/1.0.0"
    default_options = {"boost/*:shared": True, "qt/*:shared": True, "qwt/*:shared": True}
    generators = "CMakeDeps"
 
    # place the generator files under the build folder
    def layout(self):
        self.folders.generators = "build"
 
    # copy the libs to build/bin and build/lib folders
    def generate(self):
        for dep in self.dependencies.values():
            if self.settings.compiler == "apple-clang":
                 copy(self, "*.dylib", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.source_folder, "build/lib"), keep_path=False)
            elif self.settings.compiler == "gcc":
                 copy(self, "*.so", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.source_folder, "build/lib"), keep_path=False)
            elif self.settings.compiler == "msvc":
                 copy(self, "*.lib", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.source_folder,"build/lib"), keep_path=False)
                 copy(self, "*.dll", src=dep.cpp_info.bindirs[0], dst=os.path.join(self.source_folder,"build/bin"), keep_path=False)


    def get_platform(self):
        return os.environ.get('PLATFORM', None)

[leinadme]

I figured it out while dong the c/p. Can you see what I did wrong?
I have the following:
def layout(self):
self.folders.generators = "build"

but when I do the copy, I used this.
copy(self, "*.lib", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.source_folder,"build/lib"), keep_path=False)

Should be, remove the 'build' from 'build/bin' since build was already defined :)
copy(self, "*.lib", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.source_folder,"bin"), keep_path=False

[memsharded]

Great 😄

I was about to test it, but got stuck in:

ERROR: Package 'sqliteinterface/1.1.0' not resolved: Unable to find 'sqliteinterface/1.1.0' in remotes

Maybe it is your private package?

In any case, happy you found the issue. I'd like to clarify that maybe it makes sense to use dst=os.path.join(self.build_folder, "lib"), instead of using self.source_folder as the destination. Because if you are working locally or something, there is some risk that you would be polluting the sources, while typically this copy is put together with the "build" artifacts. It is a small detail, because as you didn't define self.folders.source nor self.folders.build they will be the same and default to "." (or the recipe folder), but in other cases that you might define self.folders.build = "build" instead, as some defaults like cmake_layout() do, then it would be better to have the copy() copy there.

[leinadme]

I'm sorry for driving you crazy, but as it turns out I still have the same issue. Seems that I didn't clean up properly after retrying so what I had was a legacy install which I thought did work. It works fine if I put the bin and libs in the root folder but I do want to have them in the build folder.
You are correct, the sqliteinterface is a private library but you can skip that to try to recreate the issue, if you have time.

Basically
root folder
------build folder
-----------bin
-----------lib
-----------cmake-config files

Thanks!

[leinadme]

Ok, now I think I have it :)
The name of the game is layouts.

def layout(self):
self.folders.generators = "build"
self.cpp.build.libdirs = "lib" # write the .libs to the library folder under build
self.cpp.build.bindirs = "bin" # write the .dll to the bin folder under build

# copy the libs to build/bin and build/lib folders
def generate(self):
    for dep in self.dependencies.values():
        if self.settings.compiler == "apple-clang":
             copy(self, "*.dylib", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.cpp.build.libdirs, ""), keep_path=False)
        elif self.settings.compiler == "gcc":
             copy(self, "*.so", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.cpp.build.libdirs, ""), keep_path=False)
        elif self.settings.compiler == "msvc":
             copy(self, "*.lib", src=dep.cpp_info.libdirs[0], dst=os.path.join(self.cpp.build.libdirs, ""), keep_path=False)  
             copy(self, "*.dll", src=dep.cpp_info.bindirs[0], dst=os.path.join(self.cpp.build.bindirs, ""),keep_path=False)

[memsharded]

I was trying it, but then I hit some "missing" binary, so it took me a while to build dependencies from source.

I think it would be clarifying in these cases to print the folders to understand what is happening:

    def layout(self):
        self.folders.generators = "build"
        self.cpp.build.libdirs = "lib" # write the .libs to the library folder under build
        self.cpp.build.bindirs = "bin" # write the .dll to the bin folder under build

    # copy the libs to build/bin and build/lib folders
    def generate(self):
        print("CWD", os.getcwd())
        print("SOURCE", self.source_folder)
        print("BUILD", self.build_folder)
        print("LIBDIRS", self.cpp.build.libdirs)
        print("BINDIRS", self.cpp.build.bindirs)

If we execute this, we get:

conanfile.py (my-app/1.5.4): Calling generate()                                 
conanfile.py (my-app/1.5.4): Generators folder: C:\Users\memsharded\conanws\kk\build 
CWD C:\Users\memsharded\conanws\kk\build                                             
SOURCE C:\Users\memsharded\conanws\kk                                                
BUILD C:\Users\memsharded\conanws\kk                                                 
LIBDIRS lib                                                                     
BINDIRS bin                                                                     

So we can see that your above works because the libdirs and bindirs are relative paths, and it happens that generate() runs in the generators folder, which is the "build" one.

Take into account that generators folder depending on the build system might be 1 folder both for Release and Debug configurations, if you want to have the Visual Studio IDE and be able to switch Debug/Release in the IDE itself. But the artifacts would be different, because you might collect libs and bins for Debug and for Release, and you don't want them in the same folder. This is some advanced multi-config layout for Visual Studio that cmake_layout() already implements, maybe you don't need it right now, but keep it in mind for the future.

More important is that if at some point you add a build() method, self.cpp.build.libdirs will not be pointing to that folder, you would need to define self.folders.build = "build" in the layout too.

Also the dst=os.path.join(self.cpp.build.libdirs, "") can be simplified to just dst=self.cpp.build.libdirs

Thanks for the feedback, I hope the issue is now clear and everything works! :)

[leinadme]

Thanks a bunch for your help!