Various improvements to `ament_python_install_package` #372

rhaschke · 2022-01-06T15:29:22Z

This PR implements various improvements to ament_python_install_package:

Make SCRIPTS_DESTINATION argument optional as suggested in Make ament_python_install_package() install console_scripts #328 (comment)
By using install(CODE ...) to directly generate into the target location we can still avoid creating an empty target folder.
Generating into build folder and (potentially) creating a symlink from this build to the install folder, wasn't good practice anyway.
Support multiple SCRIPTS_DESTINATIONs
This is, for example, required by xacro, which installs to lib/xacro and bin

If this got merged, I will create backport PRs for Foxy and Galactic.

The scripts are generated into a build folder and thus *always* need to be copied. No need to distinguish between symlink and other installs. Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

This avoids the problem of empty install folders as setup.py generates the target folder only if needed. Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

For example, xacro is install into lib/xacro/xacro and bin/xacro. Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

rhaschke · 2022-01-19T18:56:10Z

@hidmic, @jacobperron, could you please review? This is blocking ros/xacro#304. Thanks a lot in advance.

hidmic

Need to test this locally, but overall LGTM.

hidmic · 2022-02-07T22:50:22Z

ament_cmake_python/cmake/ament_python_install_package.cmake

-    )
+  if(POLICY CMP0087)
+    # Allow generator expressions in install(CODE...)
+    cmake_policy(SET CMP0087 NEW)


@rhaschke which generator expressions?

We don't use generator expressions, but cmake complains about the unset CMP otherwise.

The warning is even reported by CI: https://ci.ros2.org/job/ci_linux/16094/cmake/NORMAL

Hmm, I wonder if a generator expression is introduced by build_dir. Anyways, we need to cap that warning.

Yes, that's what I did. How do you trigger Full CI on this PR?

How do you trigger Full CI on this PR?

Using our ci_launcher with a custom ros2.repos file (see https://ci.ros2.org/job/ci_launcher/9781).

Yes, that's what I did.

What's funny is that I run CI yesterday and there have not been changes to this PR in weeks. Maybe fe6b3a3 was not enough?

I'm really puzzled. Indeed, the previous builds didn't show the warning - although install(CODE ...) was already used:

ament_cmake/ament_cmake_python/cmake/ament_python_install_package.cmake

Lines 190 to 191 in 3fc4db9

# compile Python files

install(CODE

However, locally, the suggested change exactly silences the warning for me. I have no clue, why it fails on CI.
My suggestion is to drop it and retry.

hidmic · 2022-02-07T22:53:19Z

Full CI, to be safe:

Linux
Linux-aarch64
Windows

This reverts commit fe6b3a3. Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

hidmic · 2022-02-08T16:32:47Z

Alright, let's give it another shot on Linux only for speed:

Linux

rhaschke · 2022-02-08T19:32:28Z

The ROS2 build farm confirms that the suggested change is required:
https://build.ros2.org/job/Rpr__ament_cmake__ubuntu_focal_amd64/173

I still have no clue why your full CI fails anyway. How should we proceed now?

hidmic · 2022-02-08T23:09:46Z

Hmm, it's just a hunch but I suspect this is an issue with CMake policy scopes.

If you look at the console output for, say, full Linux CI, before cc7ad4e, you'll see that most of the packages that generate warnings are packages that build interfaces.

That's an important detail because ament_python_install_package() gets invoked deep inside the rosidl machinery for Python. None of this will show in the local Rpr job because it only builds the packages in this repository.

Could it be that by the time your install code runs, CMP0087 has been popped off the stack?

rhaschke · 2022-02-09T01:30:51Z

Looking into the cmake traces of an old and new build of lifecycle_msgs, I'm even more confused:
Often cmake_policy(VERSION 2.6) is called (push/pop policy), essentially unsetting CMP0087, which was introduced in 3.14 only.
However, cmake doesn't complain about install(code ...) usage in this case. Why??

sloretz · 2022-04-04T23:17:08Z

ament_cmake_python/cmake/ament_python_install_package.cmake

+  # generate/install entry-point console scripts
+  foreach(_dest ${ARG_SCRIPTS_DESTINATION})
+    get_filename_component(ABS_SCRIPTS_DESTINATION "${_dest}" ABSOLUTE BASE_DIR "${CMAKE_INSTALL_PREFIX}")
+    install(CODE "execute_process(COMMAND ${python_interpreter} setup.py install_scripts --install-dir \"${ABS_SCRIPTS_DESTINATION}\"


${python_interpreter} comes from

ament_cmake/ament_cmake_python/cmake/ament_python_install_package.cmake

Line 139 in 2bf27ce

get_executable_path(python_interpreter Python3::Interpreter BUILD)

The BUILD argument will result in the path being a generator expression:

ament_cmake/ament_cmake_core/cmake/core/get_executable_path.cmake

Line 61 in 2bf27ce

set(output_var "$<TARGET_PROPERTY:${target_or_path},LOCATION>")

A workaround to make the install(CODE warnings go away is to call get_executable_path again with the CONFIGURE arugment.

get_executable_path(python_interpreter_config Python3::Interpreter CONFIGURE) # ... install(CODE "execute_process(COMMAND ${python_interpreter} setup.py ...

The same workaround is already used a little lower in the file

ament_cmake/ament_cmake_python/cmake/ament_python_install_package.cmake

Lines 189 to 197 in 2bf27ce

get_executable_path(python_interpreter_config Python3::Interpreter CONFIGURE)

# compile Python files

install(CODE

"execute_process(

COMMAND

\"${python_interpreter_config}\" \"-m\" \"compileall\"

\"${CMAKE_INSTALL_PREFIX}/${ARG_DESTINATION}/${package_name}\"

)"

)

Thanks a lot for this valuable hint. I adapted the code accordingly - sorry for the long delay.
Should be ready for merging now.

Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

rhaschke · 2023-02-27T20:32:08Z

Is there any chance that this PR will get merged?

rhaschke added 5 commits January 6, 2022 16:34

Always use USE_SOURCE_PERMISSIONS

d6c749a

The scripts are generated into a build folder and thus *always* need to be copied. No need to distinguish between symlink and other installs. Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

Make SCRIPTS_DESTINATION argument optional

a786aa2

Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

Directly generate scripts into install location

321105d

This avoids the problem of empty install folders as setup.py generates the target folder only if needed. Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

Silence policy CMP0087 warning

fe6b3a3

Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

Support multiple SCRIPTS_DESTINATIONs

d29798e

For example, xacro is install into lib/xacro/xacro and bin/xacro. Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

rhaschke force-pushed the fix-python-scripts-install branch from e01c5ab to d29798e Compare January 6, 2022 15:34

clalancette assigned hidmic Feb 3, 2022

hidmic approved these changes Feb 7, 2022

View reviewed changes

Revert "Silence policy CMP0087 warning"

cc7ad4e

This reverts commit fe6b3a3. Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

rhaschke force-pushed the fix-python-scripts-install branch from 39bab98 to cc7ad4e Compare February 8, 2022 14:38

sloretz reviewed Apr 4, 2022

View reviewed changes

audrow changed the base branch from master to rolling June 28, 2022 14:14

Use python_interpreter_config to silence CMP0087 warning

30db879

Signed-off-by: Robert Haschke <rhaschke@techfak.uni-bielefeld.de>

rhaschke force-pushed the fix-python-scripts-install branch from 04c9f78 to 30db879 Compare September 10, 2022 10:18

rhaschke mentioned this pull request Oct 21, 2022

install xacro using console_scripts entrypoint ros/xacro#304

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various improvements to `ament_python_install_package` #372

Various improvements to `ament_python_install_package` #372

rhaschke commented Jan 6, 2022 •

edited

rhaschke commented Jan 19, 2022

hidmic left a comment

hidmic Feb 7, 2022

rhaschke Feb 8, 2022

rhaschke Feb 8, 2022

hidmic Feb 8, 2022

rhaschke Feb 8, 2022

hidmic Feb 8, 2022

rhaschke Feb 8, 2022

hidmic commented Feb 7, 2022

hidmic commented Feb 8, 2022 •

edited

rhaschke commented Feb 8, 2022

hidmic commented Feb 8, 2022 •

edited

rhaschke commented Feb 9, 2022

sloretz Apr 4, 2022

rhaschke Sep 10, 2022

rhaschke commented Feb 27, 2023

	get_executable_path(python_interpreter_config Python3::Interpreter CONFIGURE)
	# compile Python files
	install(CODE
	"execute_process(
	COMMAND
	\"${python_interpreter_config}\" \"-m\" \"compileall\"
	\"${CMAKE_INSTALL_PREFIX}/${ARG_DESTINATION}/${package_name}\"
	)"
	)

Various improvements to ament_python_install_package #372

Are you sure you want to change the base?

Various improvements to ament_python_install_package #372

Conversation

rhaschke commented Jan 6, 2022 • edited

rhaschke commented Jan 19, 2022

hidmic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hidmic commented Feb 7, 2022

hidmic commented Feb 8, 2022 • edited

rhaschke commented Feb 8, 2022

hidmic commented Feb 8, 2022 • edited

rhaschke commented Feb 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhaschke commented Feb 27, 2023

Various improvements to `ament_python_install_package` #372

Various improvements to `ament_python_install_package` #372

rhaschke commented Jan 6, 2022 •

edited

hidmic commented Feb 8, 2022 •

edited

hidmic commented Feb 8, 2022 •

edited