Feature/package id mode (#4958)

* remotes to metadata

* WIP

* layout of dev

* wip

* Removed versions

* working but not tested

* local

* Local, package failing because of none timestamp in PREV

* WIP

* search tests ok

* Fixing tests and small refactor

* renamed module

* review

* ISO in client, reviewed

* Fixing tests

* Fixing tests

* New tests about metadata being updated

* Review

* Time cleared on export/package

* Fix revisions list

* fixed test

* Unused import

* WIP

* Review

* Fixed test message assert

* Use env var instead of test client

* Cool testing framework and a lot of remove with revisions tests

* Remove tests revisions

* Drafting tests

* Test passing, a lot missing yet

* PENDING MANY TESTS

* some more tests

* Pending many tests

* Added more tests

* No revisions tests passing

* make BinariesAnalyzer the one responsible for the IDs

* remove build mode check

* Tests passing, only about 10 to complete

* changing how binary IDs are computed

* fixing tests

* refactoring tests

* binary ID belongs to graph Node

* working in graph

* Finished tests

* Upload package id not reference

* One more test

* SVN test

* IOError

* Fixing tests

* Fixing windows test with export package and package revisions

* Fixed test

* Pending migration

* Migration OK

* working...

* renaming conan->recipe

* Revisioning the revisions pull request

* CONAN_REVISIONS_ENABLED

* WIP

* Safer migration

* No revisions test

* V2 WIP

* WIP

* Review

* working except full_requires test broken

* WIP

* Rest routes

* working....

* working on it, very bad state

* Refactor routes

* Passing wihout revisions

* Removed time from metadata

* working but ConanInfo.full_requires test

* Passing tests

* Fixed more tests

* binary analyzer refactor

* Self review

* Imports

* small optimization of set_dirty(package_folder)

* Fixed rest api test

* Fixed slows

* Fixed py2 issue with exception translation with http codes

* rest api prepared revisions

* new graph builder

* working

* fixed test

* Dani and Diego review

* tests passing

* Fixing tests

* working

* Review

* Imports

* working...

* renamed bid->package_id

* import and comment

* renamed test

* tests passing

* changes

* removed prints

* moving tests

* improving tests

* new tests

* fixed py3

* working

* package-ID modes

* fixing tests

* fixing build_id test

* cleaning

* full test of package_id_mode

* package_id config modes

* fixing things

* working

* getting there...

* tests passing but RegistryTest and RemoteTest

* python requires updates

* fixed test with time.sleep()

* working

* adding definition of remotes too

* working in dumps, new remotes.txt file?

* tests passing missing file format

* fixing tests

* working

* remove simple paths

* remove simple paths

* renames

* removed print

* abs_path for CONAN_USER_HOME too

* fix storage_path

* working, not ready for review

* working...

* working..

* working...

* review and migrations for conan-config-install

* fix test

* fix error

* first draft ready

* added build-requires (forced for Unknown) capability

* refactor GraphBinariesAnalyzer

* fixing things

* allowing updating lock for PACKAGE_ID_UNKNOWN

* 1.19.0 release

* 1.20.0-dev

* Issue/5814 fix python_requires with short_paths enabled (#5841)

* Added required = True to subparsers in order to print error message in Py2 and Py3.

* sync

* basic concurrent upload at reference level with futures

* revert changes

* add line

* Lock buggy urllib3 (#5808)

* app simplifying (#5806)

* Apply lockfile before updating downstream requires (#5771)

* apply graph_lock before looking for overrides

* first step: get rid of the warning

* cleaner if graph_lock is passed to the function

* only update requires upstream if no lockfile is applied

* fix tests

* Deprecation of CONAN_USERNAME and CONAN_CHANNEL: fix error message (#5756)

* if CONAN_USERNAME and CONAN_CHANNEL are deprecated, the error cannot recommend them

* update tests accordingly

* test client load() file method (#5815)

* no user/channel repr without _ (#5817)

* no user/channel repr without _

* minor fixes

* fix tests

* Remove py34 (#5820)

* fix upload package id (#5824)

* - update macOS, watchOS, tvOS, iOS version numbers (#5823)

* Refresh token client support.  (#5662)

* Refresh token client support. Missing tests. Missing migration

* public method

* WIP

* Refresh almost there

* Removed prints

* Try migrate

* Migration

* Add comment

* Refresh token flow following RFC recommentations

* Refresh ok

* review

* Remove traces

* Refactor capabilities

* Removed tmp file

* Review

* #5819 Show warning message for Python 3.4 (#5829)

* #5819 Show warning message for Python 3.4

- Add new warning message for python 3.4 which is no longer supported
- Added funcional tests to validate both python 3.4 and 2.x

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #5819 Fix broken tests

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* Add cpp_info.name to cmake and pkg_config generators (#5598)

* Add cpp_info.name to cmake generators

* Fix unit tests to mimic real behavior

* cmake_paths test

* add test for cmake generator

* Add cmake_find_package test

* fix test in py3

* Applied cpp_info.name to pkg_config generator

* check different name in pkg_config

* use pyreq short path if exists

* sync with develop

* revert change

* pass correct conanfile

* add test

* remove line

* clean test

* Feature/remote enable error handling (#5835)

* Add a test

* Raise an error if no remote found

* Check that we don't raise an error if the state is already in the required state

* Raise an error only if the remote name is not a wildcard

* Check that we don't raise an error with a wildcard if there are no remotes

* Removed go test files and references to golang from README (#5854)

* Removed go test files and references to golang from README

* Removed golang from README

* making GraphBinaryAnalyzer injectable

* merged

* minor changes

* different approach to recompute ID

* unused import

* minor code structure changes

* fix test!

* message and reduced test

* tests with revisions

conans/client/conan_api.py

@@ -478,7 +478,8 @@ def workspace_install(self, path, settings=None, options=None, env=None,
                     node.conanfile.generators = tmp
 
         installer = BinaryInstaller(self.app, recorder=recorder)
-        installer.install(deps_graph, remotes, keep_build=False, graph_info=graph_info)
+        installer.install(deps_graph, remotes, build, update, keep_build=False,
+                          graph_info=graph_info)
 
         install_folder = install_folder or cwd
         workspace.generate(install_folder, deps_graph, self.app.out)

conans/client/graph/graph_binaries.py

@@ -4,8 +4,7 @@
                                        BINARY_SKIP, BINARY_UPDATE,
                                        RECIPE_EDITABLE, BINARY_EDITABLE,
                                        RECIPE_CONSUMER, RECIPE_VIRTUAL)
-from conans.errors import NoRemoteAvailable, NotFoundException, \
-    conanfile_exception_formatter
+from conans.errors import NoRemoteAvailable, NotFoundException, conanfile_exception_formatter
 from conans.model.info import ConanInfo, PACKAGE_ID_UNKNOWN
 from conans.model.manifest import FileTreeManifest
 from conans.model.ref import PackageReference
@@ -154,13 +153,15 @@ def _evaluate_is_cached(self, node, pref):
     def _evaluate_node(self, node, build_mode, update, remotes):
         assert node.binary is None, "Node.binary should be None"
         assert node.package_id is not None, "Node.package_id shouldn't be None"
+        assert node.package_id != PACKAGE_ID_UNKNOWN, "Node.package_id shouldn't be Unknown"
         assert node.prev is None, "Node.prev should be None"
 
         if node.package_id == PACKAGE_ID_UNKNOWN:
             node.binary = BINARY_MISSING
             return
 
         ref, conanfile = node.ref, node.conanfile
+
         # If it has lock
         locked = node.graph_lock_node
         if locked and locked.pref.id == node.package_id:
@@ -219,27 +220,46 @@ def _evaluate_node(self, node, build_mode, update, remotes):
         node.binary_remote = remote
 
     @staticmethod
-    def _compute_package_id(node, default_package_id_mode):
+    def _propagate_options(node):
+        # TODO: This has to be moved to the graph computation, not here in the BinaryAnalyzer
+        # as this is the graph model
         conanfile = node.conanfile
         neighbors = node.neighbors()
-        direct_reqs = []  # of PackageReference
-        indirect_reqs = set()   # of PackageReference, avoid duplicates
+        transitive_reqs = set()  # of PackageReference, avoid duplicates
         for neighbor in neighbors:
             ref, nconan = neighbor.ref, neighbor.conanfile
-            direct_reqs.append(neighbor.pref)
-            indirect_reqs.update(nconan.info.requires.refs())
+            transitive_reqs.add(neighbor.pref)
+            transitive_reqs.update(nconan.info.requires.refs())
+
             conanfile.options.propagate_downstream(ref, nconan.info.full_options)
             # Might be never used, but update original requirement, just in case
             conanfile.requires[ref.name].ref = ref
 
-        # Make sure not duplicated
-        indirect_reqs.difference_update(direct_reqs)
-        # There might be options that are not upstream, backup them, might be
-        # for build-requires
+        # There might be options that are not upstream, backup them, might be for build-requires
         conanfile.build_requires_options = conanfile.options.values
-        conanfile.options.clear_unused(indirect_reqs.union(direct_reqs))
+        conanfile.options.clear_unused(transitive_reqs)
         conanfile.options.freeze()
 
+    @staticmethod
+    def _compute_package_id(node, default_package_id_mode):
+        """
+        Compute the binary package ID of this node
+        :param node: the node to compute the package-ID
+        :param default_package_id_mode: configuration of the package-ID mode
+        """
+        # TODO Conan 2.0. To separate the propagation of the graph (options) of the package-ID
+        # A bit risky to be done now
+        conanfile = node.conanfile
+        neighbors = node.neighbors()
+        direct_reqs = []  # of PackageReference
+        indirect_reqs = set()  # of PackageReference, avoid duplicates
+        for neighbor in neighbors:
+            ref, nconan = neighbor.ref, neighbor.conanfile
+            direct_reqs.append(neighbor.pref)
+            indirect_reqs.update(nconan.info.requires.refs())
+
+        # Make sure not duplicated
+        indirect_reqs.difference_update(direct_reqs)
         conanfile.info = ConanInfo.create(conanfile.settings.values,
                                           conanfile.options.values,
                                           direct_reqs,
@@ -270,8 +290,30 @@ def _handle_private(self, node):
     def evaluate_graph(self, deps_graph, build_mode, update, remotes):
         default_package_id_mode = self._cache.config.default_package_id_mode
         for node in deps_graph.ordered_iterate():
+            self._propagate_options(node)
             self._compute_package_id(node, default_package_id_mode)
             if node.recipe in (RECIPE_CONSUMER, RECIPE_VIRTUAL):
                 continue
+            if node.package_id == PACKAGE_ID_UNKNOWN:
+                assert node.binary is None
+                continue
             self._evaluate_node(node, build_mode, update, remotes)
             self._handle_private(node)
+
+    def reevaluate_node(self, node, remotes, build_mode, update):
+        assert node.binary is None
+        output = node.conanfile.output
+        node._package_id = None  # Invalidate it, so it can be re-computed
+        default_package_id_mode = self._cache.config.default_package_id_mode
+        output.info("Unknown binary for %s, computing updated ID" % str(node.ref))
+        self._compute_package_id(node, default_package_id_mode)
+        output.info("Updated ID: %s" % node.package_id)
+        if node.recipe in (RECIPE_CONSUMER, RECIPE_VIRTUAL):
+            return
+        assert node.package_id != PACKAGE_ID_UNKNOWN
+        self._evaluate_node(node, build_mode, update, remotes)
+        output.info("Binary for updated ID from: %s" % node.binary)
+        if node.binary == BINARY_BUILD:
+            output.info("Binary for the updated ID has to be built")
+        else:
+            output.info("Binary for the updated ID from: %s" % node.binary)

conans/client/graph/graph_manager.py

@@ -14,6 +14,7 @@
 from conans.model.ref import ConanFileReference
 from conans.paths import BUILD_INFO
 from conans.util.files import load
+from conans.model.info import PACKAGE_ID_UNKNOWN
 
 
 class _RecipeBuildRequires(OrderedDict):
@@ -230,8 +231,9 @@ def _recurse_build_requires(self, graph, subgraph, builder, check_updates,
             # FIXME: To be improved and build a explicit model for this
             if node.recipe == RECIPE_VIRTUAL:
                 continue
+            # Packages with PACKAGE_ID_UNKNOWN might be built in the future, need build requires
             if (node.binary not in (BINARY_BUILD, BINARY_EDITABLE)
-                    and node.recipe != RECIPE_CONSUMER):
+                    and node.recipe != RECIPE_CONSUMER and node.package_id != PACKAGE_ID_UNKNOWN):
                 continue
             package_build_requires = self._get_recipe_build_requires(node.conanfile)
             str_ref = str(node.ref)

conans/client/installer.py

@@ -20,6 +20,7 @@
 from conans.model.editable_layout import EditableLayout
 from conans.model.env_info import EnvInfo
 from conans.model.graph_info import GraphInfo
+from conans.model.info import PACKAGE_ID_UNKNOWN
 from conans.model.manifest import FileTreeManifest
 from conans.model.ref import PackageReference
 from conans.model.user_info import UserInfo
@@ -294,17 +295,18 @@ def __init__(self, app, recorder):
         self._out = app.out
         self._remote_manager = app.remote_manager
         self._recorder = recorder
+        self._binaries_analyzer = app.binaries_analyzer
         self._hook_manager = app.hook_manager
 
-    def install(self, deps_graph, remotes, keep_build=False, graph_info=None):
+    def install(self, deps_graph, remotes, build_mode, update, keep_build=False, graph_info=None):
         # order by levels and separate the root node (ref=None) from the rest
         nodes_by_level = deps_graph.by_levels()
         root_level = nodes_by_level.pop()
         root_node = root_level[0]
         # Get the nodes in order and if we have to build them
-        self._build(nodes_by_level, keep_build, root_node, graph_info, remotes)
+        self._build(nodes_by_level, keep_build, root_node, graph_info, remotes, build_mode, update)
 
-    def _build(self, nodes_by_level, keep_build, root_node, graph_info, remotes):
+    def _build(self, nodes_by_level, keep_build, root_node, graph_info, remotes, build_mode, update):
         processed_package_refs = set()
         for level in nodes_by_level:
             for node in level:
@@ -324,12 +326,15 @@ def _build(self, nodes_by_level, keep_build, root_node, graph_info, remotes):
                         continue
                     assert ref.revision is not None, "Installer should receive RREV always"
                     _handle_system_requirements(conan_file, node.pref, self._cache, output)
+                    if node.package_id == PACKAGE_ID_UNKNOWN:
+                        self._binaries_analyzer.reevaluate_node(node, remotes, build_mode, update)
                     self._handle_node_cache(node, keep_build, processed_package_refs, remotes)
 
         # Finally, propagate information to root node (ref=None)
         self._propagate_info(root_node)
 
-    def _node_concurrently_installed(self, node, package_folder):
+    @staticmethod
+    def _node_concurrently_installed(node, package_folder):
         if node.binary == BINARY_DOWNLOAD and os.path.exists(package_folder):
             return True
         elif node.binary == BINARY_UPDATE:
@@ -376,7 +381,7 @@ def _handle_node_editable(self, node, graph_info):
     def _handle_node_cache(self, node, keep_build, processed_package_references, remotes):
         pref = node.pref
         assert pref.id, "Package-ID without value"
-
+        assert pref.id != PACKAGE_ID_UNKNOWN, "Package-ID error: %s" % str(pref)
         conanfile = node.conanfile
         output = conanfile.output
 

conans/client/manager.py

@@ -1,6 +1,7 @@
 import os
 
 from conans.client.generators import write_generators
+from conans.client.graph.build_mode import BuildMode
 from conans.client.graph.graph import RECIPE_CONSUMER, RECIPE_VIRTUAL
 from conans.client.graph.printer import print_graph
 from conans.client.importer import run_deploy, run_imports
@@ -61,7 +62,10 @@ def deps_install(app, ref_or_path, install_folder, graph_info, remotes=None, bui
         pass
 
     installer = BinaryInstaller(app, recorder=recorder)
-    installer.install(deps_graph, remotes, keep_build=keep_build, graph_info=graph_info)
+    # TODO: Extract this from the GraphManager, reuse same object, check args earlier
+    build_modes = BuildMode(build_modes, out)
+    installer.install(deps_graph, remotes, build_modes, update, keep_build=keep_build,
+                      graph_info=graph_info)
     # GraphLock always != None here (because of graph_manager.load_graph)
     graph_info.graph_lock.update_check_graph(deps_graph, out)
 

conans/model/graph_lock.py

@@ -5,6 +5,7 @@
     BINARY_BUILD
 from conans.client.profile_loader import _load_profile
 from conans.errors import ConanException
+from conans.model.info import PACKAGE_ID_UNKNOWN
 from conans.model.options import OptionsValues
 from conans.model.ref import PackageReference, ConanFileReference
 from conans.util.files import load, save
@@ -237,7 +238,7 @@ def update_check_graph(self, deps_graph, output):
                 node_pref = node.pref.copy_clear_revs() if not self.revisions_enabled else node.pref
                 # If the update is compatible (resolved complete PREV) or if the node has
                 # been build, then update the graph
-                if pref.is_compatible_with(node_pref) or \
+                if pref.id == PACKAGE_ID_UNKNOWN or pref.is_compatible_with(node_pref) or \
                         node.binary == BINARY_BUILD or node.id in affected:
                     lock_node.pref = node.pref
                 else:

conans/model/options.py

@@ -610,10 +610,10 @@ def propagate_downstream(self, ref, options):
         for k, v in options._reqs_options.items():
             self._deps_package_values[k] = v.copy()
 
-    def clear_unused(self, references):
+    def clear_unused(self, prefs):
         """ remove all options not related to the passed references,
         that should be the upstream requirements
         """
-        existing_names = [r.ref.name for r in references]
+        existing_names = [pref.ref.name for pref in prefs]
         self._deps_package_values = {k: v for k, v in self._deps_package_values.items()
                                      if k in existing_names}

conans/test/functional/command/upload_test.py

@@ -16,6 +16,7 @@
 from conans.test.utils.cpp_test_files import cpp_hello_conan_files
 from conans.test.utils.tools import NO_SETTINGS_PACKAGE_ID, TestClient, TestServer, \
     TurboTestClient, GenConanfile
+from conans.util.env_reader import get_env
 from conans.util.files import gzopen_without_timestamps, is_dirty, save
 
 conanfile = """from conans import ConanFile
@@ -621,6 +622,7 @@ def upload_login_prompt_disabled_user_authenticated_test(self):
         self.assertIn("Uploading conanfile.py", client.out)
         self.assertIn("Uploading conan_export.tgz", client.out)
 
+    @unittest.skipUnless(get_env("TESTING_REVISIONS_ENABLED", False), "Only revisions")
     def upload_key_error_test(self):
         files = cpp_hello_conan_files("Hello0", "1.2.1", build=False)
         server1 = TestServer([("*/*@*/*", "*")], [("*/*@*/*", "*")], users={"lasote": "mypass"})
@@ -630,7 +632,6 @@ def upload_key_error_test(self):
         servers["server2"] = server2
         client = TestClient(servers=servers)
         client.save(files)
-        client.run("config set general.revisions_enabled=True")
         client.run("create . user/testing")
         client.run("user lasote -p mypass")
         client.run("user lasote -p mypass -r server2")