[bug] the “exported” files are not copied to the self.source_folder before calling the source() method

[nathan-b-flya]

Describe the bug

According to the latest conan documentation (v 2.2) https://docs.conan.io/2.2/reference/conanfile/methods/source.html:

The source() method should not access nor manipulate files in other folders different to the self.source_folder. All the “exported” files are copied to the self.source_folder before calling it.

But the exported files (in the exports attribute) don't seem to be copied in the source folder before the source() method is called.

How to reproduce it

Conan version: 2.2.2
Python version: 3.8.10
OS: Ubuntu 20

In an empty directory:

conanfile.py:

from conan import ConanFile

class BugReportConan(ConanFile):
    name = "bug_report"

    exports = ["foo.repos",]

    def source(self):
        self.output.info("============= 1")
        self.output.info(self.source_folder)
        self.output.info(self.export_folder)
        self.output.info(self.recipe_folder)
        with open("foo.repos") as stream:
            pass

foo.repos (can be anything):

repositories:
  foo/bar:
    type: git
    url: https://github.com/foo/bar.git
    version: 0.0.0

Calling:
conan create . --version 0.0.0
gives an error that the foo.repos does not exist:

$ conan create . --version 0.0.0
======== Exporting recipe to the cache ========
bug_report/0.0.0: Exporting package recipe: /home/nathan/flyability/tmp/flya_conan-center-index/bug/conanfile.py
bug_report/0.0.0: Copied 1 '.py' file: conanfile.py
bug_report/0.0.0: Copied 1 '.repos' file: foo.repos
bug_report/0.0.0: Exported to cache folder: /home/nathan/.conan2/p/bug_re7a009f232436/e
bug_report/0.0.0: Exported: bug_report/0.0.0#db2cd895f4402fa553e3e67078d50209 (2024-04-05 09:07:39 UTC)

======== Input profiles ========
[...]
======== Computing dependency graph ========
[...]
======== Computing necessary packages ========
[...]
======== Installing packages ========
bug_report/0.0.0: Calling source() in /home/nathan/.conan2/p/bug_re7a009f232436/s
bug_report/0.0.0: ============= 1
bug_report/0.0.0: /home/nathan/.conan2/p/bug_re7a009f232436/s
bug_report/0.0.0: None
bug_report/0.0.0: /home/nathan/.conan2/p/bug_re7a009f232436/e
ERROR: bug_report/0.0.0: Error in source() method, line 13
	with open("foo.repos") as stream:
	FileNotFoundError: [Errno 2] No such file or directory: 'foo.repos'

Indeed, the file foo.repos is correctly exported in the export folder, but the source folder s/ is empty:

$ tree -F /home/nathan/.conan2/p/bug_re7a009f232436/
/home/nathan/.conan2/p/bug_re7a009f232436/
├── d/
│   └── metadata/
├── e/
│   ├── conanfile.py
│   ├── conanmanifest.txt
│   └── foo.repos
├── es/
├── s/
└── s.dirty

5 directories, 4 files

And weirdly the self.export_folder is None and the self.recipe_folder is the folder where the files are exported (/home/nathan/.conan2/p/bug_re7a009f232436/e)

Note: I tested the same example with conan==1.63.0 using conan create . bug_report/0.0.0@ and it works correctly (i.e. the source folder contains the foo.repos file)

[memsharded]

Hi @nathan-b-flya

Thanks for your report.
I'd say this is not a bug, but it is true that it might benefit some clarification in the docs.

• The exports files are those that belong to the recipe conanfile.py, that is, the conanfile needs them to load. Like conandata.yml, and some other .py files that the conanfile.py might import.
• The export files are not "source", and are not copied to the source folder, they remain in the "export" folder
• The exports_sources files are the ones that are "source" and will be copied to the "source" folder

The problem is that we often refer as "exported" files to the "exported source" files.
Please change your declaration to exports_sources and let me know, and then we might want to move this to the docs repo for clarification. Thanks!

Note: it is indeed possible that it was behaving this way in Conan 1.X, but mostly as a undesired behavior that wasn't fixed back then because of the risk of breaking. Conan 2 fixed that.

[nathan-b-flya]

Thanks @memsharded for your quick answer!

Indeed, as your suggestion, replacing exports = ["foo.repos",] by exports_sources = ["foo.repos",] fixes the issue.

The exports files are those that belong to the recipe conanfile.py, that is, the conanfile needs them to load. Like conandata.yml, and some other .py files that the conanfile.py might import.

In my use case, the foo.repos is a file that the conanfile needs (similarly to the conandata.yml) (the goal is to clone some repo using vcstool, like vcs import < foo.repos), that's why I used exports attribute, and not exports_sources (which for me was to copy source dirs/files like include/*, CMakeLists.txt, etc.).

FYI, I checked on the conan-center-index, and I noticed there are two recipes relying on the exports attribute for a file (submoduledata.yml) needed by the source() method:

• https://github.com/conan-io/conan-center-index/blob/dd1d173ad/recipes/open62541/all/conanfile.py#L241-L242
• https://github.com/conan-io/conan-center-index/blob/dd1d173ad/recipes/yojimbo/all/conanfile.py#L38

It seems to work for them because they use the self.recipe_folder using os.path.join(self.recipe_folder, 'submoduledata.yml') (which is not allowed by the documentation of source() method).

So maybe, as you suggest, the doc might benefit some clarification for the exports attribute, where it seems to me the only real use case is to define some python module (like helpers.py), but not others files (like info.txt as per documentation or whatever *.repos/submoduledata.yml files).

[memsharded]

Indeed, as your suggestion, replacing exports = ["foo.repos",] by exports_sources = ["foo.repos",] fixes the issue.

Great, happy to hear that.

In my use case, the foo.repos is a file that the conanfile needs (similarly to the conandata.yml) (the goal is to clone some repo using vcstool, like vcs import < foo.repos), that's why I used exports attribute, and not exports_sources (which for me was to copy source dirs/files like include/*, CMakeLists.txt, etc.).

Sounds good. I think that if you needed for both, you can both exports and exports_sources the same file. But probably referring and copying it via recipe_folder would be preferred.

It seems to work for them because they use the self.recipe_folder using os.path.join(self.recipe_folder, 'submoduledata.yml') (which is not allowed by the documentation of source() method).

I see in https://docs.conan.io/2/reference/conanfile/methods/source.html#source, the comment:
The source() method should not access nor manipulate files in other folders different to the self.source_folder. All the “exported” files are copied to the self.source_folder before calling it. was mostly intended to protect the modification of the recipe and the creation of files in build/package folders. Reading some file in the recipe_folder should be pretty harmless.

I also think that those recipes could benefit and be simplified by allowing them to use the built-in conandata.yml, which was blocked in the past in ConanCenter due to policies, but this will be allowed in the future.

Moving this ticket to the docs repo for clarifications there. Thanks for your feedback again!