[question] How to build node from source (autotools, make)

[peakschris]

What is your question?

I am attempting to automate build of nodejs from sources, as part of our company's build chain. This package uses AutoTools and make. My current conanfile is:

import os
import shutil

from conan import ConanFile
from conan.tools.gnu import AutotoolsToolchain, Autotools, AutotoolsDeps
from conan.tools.layout import basic_layout
from conan.tools.files import chdir, download, unzip, check_sha1, rm

class nodeConan(ConanFile):
    name = "node"
    version = "v18.20.2"
    package_type = "application"

    # Optional metadata
    description = "Node.js is an open-source, cross-platform JavaScript runtime environment."
    license = "MIT"
    homepage = "https://nodejs.org"

    # Binary configuration
    settings = "os", "compiler", "build_type", "arch"

    # Sources are located in the same place as this recipe, copy them to the recipe
    exports_sources = "configure.ac", "Makefile.am", "*"

    def source(self):
        zip_name = f"node-{self.version}.tar.gz"
        # Immutable source .zip
        download(self, f"https://nodejs.org/dist/{self.version}/{zip_name}", zip_name)
        # Recommended practice, always check hashes of downloaded files
        check_sha1(self, zip_name, "e231c65c504a083654538cd368338f81e298311f")
        unzip(self, zip_name, strip_root=True)
        os.unlink(zip_name)

    def layout(self):
        basic_layout(self, src_folder="src")

    def generate(self):
        deps = AutotoolsDeps(self)
        deps.generate()
        at_toolchain = AutotoolsToolchain(self, prefix=".")
        at_toolchain.update_configure_args({
            "--bindir": None,
            "--sbindir": None,
            "--libdir": "lib",
            "--includedir": None,
            "--oldincludedir": None,
        })
        at_toolchain.generate()

    def build(self):
            autotools = Autotools(self)
        autotools.configure()
        autotools.make("V=1")

    def package(self):
        autotools = Autotools(self)
        autotools.install()

This works to download the source and configure it. However, when I run: conan create . -vvv, it fails to find the Makefile:

 > "/apps/browchri/conan/p/b/node04b91877cab94/b/src/configure" --prefix=. --libdir=lib
node/v18.20.2: RUN: "/apps/browchri/conan/p/b/node04b91877cab94/b/src/configure" --prefix=. --libdir=lib
Node.js configure: Found Python 3.11.5...
INFO: configure completed successfully

node/v18.20.2: RUN: make V=1 -j40
make: *** No targets specified and no makefile found.  Stop.

node/v18.20.2: ERROR:
Package '24318d598c2aa43c2494894e6c0ea0d428fd0cff' build failed
node/v18.20.2: WARN: Build folder /apps/browchri/conan/p/b/node04b91877cab94/b/build-release
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/conans/errors.py", line 52, in conanfile_exception_formatter
    yield
  File "/usr/local/lib/python3.6/site-packages/conans/client/conanfile/build.py", line 14, in run_build_method
    conanfile.build()

There is a makefile in the src directory after running configure:

# dir /apps/browchri/conan/p/b/node04b91877cab94/b/src
BSDmakefile         GOVERNANCE.md  __pycache__           codecov.yml    configure     icu_config.gypi  out             tsconfig.json
BUILDING.md         LICENSE        android-configure     common.gypi    configure.py  lib              pyproject.toml  typings
CHANGELOG.md        Makefile       android-patches       config.gypi    deps          node.gyp         src             vcbuild.bat
CODE_OF_CONDUCT.md  README.md      android_configure.py  config.mk      doc           node.gypi        test
CONTRIBUTING.md     SECURITY.md    benchmark             config.status  glossary.md   onboarding.md    tools

There is no makefile in the build-release directory:

# dir /apps/browchri/conan/p/b/node04b91877cab94/b/build-release
conan
# dir /apps/browchri/conan/p/b/node04b91877cab94/b/build-release/conan
conanautotoolsdeps.sh       conanbuild.sh                    conanrunenv-release-x86_64.sh          deactivate_conanbuild.sh
conanautotoolstoolchain.sh  conanbuildenv-release-x86_64.sh  deactivate_conanautotoolsdeps.sh       deactivate_conanbuildenv-release-x86_64.sh
conanbuild.conf             conanrun.sh                      deactivate_conanautotoolstoolchain.sh  deactivate_conanrun.sh

It feels like I must be making a basic mistake, but having pored over all the docs, I can't figure out what this might be. Any help would be much appreciated!

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @peakschris

Thanks for your question.

I guess that you know there is a recipe for nodejs in ConanCenter in https://github.com/conan-io/conan-center-index/blob/master/recipes/nodejs/all/conanfile.py, but this one is downloading the pre-compiled binaries, and you really need to build it from sources?

Then I was trying to reproduce with your recipe and I got:

node/v18.20.2: Calling build()
node/v18.20.2: RUN: "/home/memsharded/.conan2/p/b/node1c59957719cd1/b/src/configure" --prefix=. --libdir=lib
Node.js configure: Found Python 3.10.12...
INFO: configure completed successfully

node/v18.20.2: RUN: make V=1 -j8
make: *** No targets specified and no makefile found.  Stop.

Then I realized that I didn't have the files that it seems you are export-sources, a Makefile.am and a configure.ac, but the configure step of the project didn't complain about these missing files.

Maybe the build of nodejs doesn't use a standard makefile? Maybe you need to call self.run("something-else to launch the build?
This seems a bit outside of the Conan scope, and more related to specific knowledge how to build nodejs from source, but unfortunately I don't have that expertise.

[peakschris]

Hi @memsharded thanks for your reply. I have seen the binary-only nodejs package, we have a requirement to build from source. We already have this working in a dockerfile, very straightforward:

RUN tar -xf "node-v$NODE_VERSION.tar.xz" \
    && cd "node-v$NODE_VERSION" \
    && ./configure \
    && make -j$(getconf _NPROCESSORS_ONLN) V= \
    && make install \

With Conan, I get exactly the same behaviour as you - configure works, but make does not.

I've removed exports_sources, and there is no difference.

Which current working directory does conan run make from? And isn't it curious that there is no source copied into the build folder?

[memsharded]

Ok, I see what is happening.
Let me try to clarify first: most of the things that you are using in the recipe might not be needed. So lets strip them down, and use a much simpler recipe:

class nodeConan(ConanFile):
    name = "node"
    version = "18.20.2"  # better remove the "v" works better with version ranges
    package_type = "application"

    # Optional metadata
    description = "Node.js is an open-source, cross-platform JavaScript runtime environment."
    license = "MIT"
    homepage = "https://nodejs.org"

    # Binary configuration
    settings = "os", "arch"

    def source(self):
        zip_name = f"node-v{self.version}.tar.gz"
        # Immutable source .zip
        download(self, f"https://nodejs.org/dist/v{self.version}/{zip_name}", zip_name)
        # Recommended practice, always check hashes of downloaded files
        check_sha1(self, zip_name, "e231c65c504a083654538cd368338f81e298311f")
        unzip(self, zip_name, strip_root=True)
        os.unlink(zip_name)

    def build(self):
        self.run("./configure")
        self.run("make V=1")

    def package(self):
        self.run("make install")  # missing something to target the self.package_folder

That should work fine with conan create . (except the last part, the package_folder should be passed somewhere so make install does the final install to this folder.

Some notes:

• No layout is necessary, if you don't aim to work on the source code of nodejs. That is mostly for working in the "developer flow" and editables, or to have cleaner layouts for build systems like CMake that supports them. Not really the case for autotools.
• I have dropped the compiler and build_type settings, as I am not really using them, just launching the default "make"
• I am not using the Conan integrations for autotools. There are no dependencies, so AutotoolsDeps not necessary. I might trust that configure + make will do the right thing.
• As I am dropping those settings, I cannot have multiple binaries for nodejs package built for different compilers and compiler versions, but it sounds you might not need it, as you want just the executable, the need is to build it from source.

Please let me know if this helps.

[peakschris]

@memsharded thank you for this suggestion, it looks much more straightforward.

We do need to run the build on two different machines (windows and rockylinux), and upload the resulting binaries to our internal Conan repo. configure and make without arguments will do the right thing on each of these platforms. Do we need to add anything back in to label the binaries appropriately?

Thanks!

[memsharded]

We do need to run the build on two different machines (windows and rockylinux), and upload the resulting binaries to our internal Conan repo. configure and make without arguments will do the right thing on each of these platforms. Do we need to add anything back in to label the binaries appropriately?

Most likely not, if you control well your build environments, and don't have any issues with those binaries, this might be a perfectly valid (and pragmatic) approach. The recipe models settings = "os", "arch" so it will be perfectly enough to model your binary variability.

Note I also changed the version to not include the v, as it will work with version ranges too.

What you might want to add is some package_info() like adding some env-vars or the like, so consumers that tool_requires("nodejs... will automatically get configured their environment so it is easier for them to use/call it.

[memsharded]

Any further question or feedback here?
If not, I think we can close this ticket, thanks!

[peakschris]

Yes, this input was very helpful and it's now working. Thank you!

[memsharded]

Happy to hear, thanks for the feedback and following up.