xgboost.DMatrix raises AttributeError in xgboost 1.3.0

[harupy]

Thanks for participating in the XGBoost community! We use https://discuss.xgboost.ai for any general usage questions and discussions. The issue tracker is used for actionable items such as feature proposals discussion, roadmaps, and bug tracking. You are always welcomed to post on the forum first :)

Issues that are inactive for a period of time may get closed. We adopt this policy so that we won't lose track of actionable issues that may fall at the bottom of the pile. Feel free to reopen a new one if you feel there is an additional problem that needs attention when an old one gets closed.

For bug reports, to help the developer act on the issues, please include a description of your environment, preferably a minimum script to reproduce the problem.

For feature proposals, list clear, small actionable items so we can track the progress of the change.

Code to reproduce the error.

>>> import xgboost as xgb
>>> xgb.__version__
'1.3.0'

>>> xgb.DMatrix(np.random.rand(100, 100))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.6/site-packages/xgboost/core.py", line 510, in __init__
    self.feature_names = feature_names
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.6/site-packages/xgboost/core.py", line 844, in feature_names
    _check_call(_LIB.XGDMatrixSetStrFeatureInfo(
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fa5f406a960, XGDMatrixSetStrFeatureInfo): symbol not found

[hcho3]

@harupy It appears that the XGBoost package inside your virtual env is conflicting with an older XGBoost package installed somewhere else in the system. Can you scan your system to check for other copies of XGBoost?

[hcho3]

I cannot reproduce the error on my Linux machine after installing a fresh copy of XGBoost.

[hcho3]

@harupy Usually, the XGBoost python package should locate libxgboost.so from the local Python environment first. For some reason, it's picking up another libxgboost.so somewhere else. It might be that the local Python environment does not contain the file libxgboost.so due to incomplete installation of XGBoost.

Currently, here are the list of paths libxgboost.so is searched in. They are listed in the order of highest to lowest priorities:

xgboost/python-package/xgboost/libpath.py

Lines 21 to 30 in 0ffaf0f

	curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
	dll_path = [
	# normal, after installation `lib` is copied into Python package tree.
	os.path.join(curr_path, 'lib'),
	# editable installation, no copying is performed.
	os.path.join(curr_path, os.path.pardir, os.path.pardir, 'lib'),
	# use libxgboost from a system prefix, if available. This should be the last
	# option.
	os.path.join(sys.prefix, 'lib'),
	]

[harupy]

@hcho3 Thanks for the quick reply. You're right. The error I got seems related to my environment.

[harupy]

This Dockerfile:

FROM python:3.7

RUN pip install xgboost
RUN python -c "import xgboost as xgb; import numpy as np; xgb.DMatrix(np.random.rand(100, 100))"

gave:

Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!

[hcho3]

@harupy The issue seems related to #6480. I'm actively working on it.

[hcho3]

@harupy I reproduced the issue on my end using the container python:3.7. It turns out that simply importing XGBoost will cause segmentation fault:

root@1b9c98e1a10a:/# python -c "import xgboost"
Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!

[harupy]

I was able to repro the error by importing xgboost:

FROM python:3.7

RUN pip install xgboost
RUN python -c "import xgboost as xgb"

logs

Step 1/3 : FROM python:3.7
 ---> 7fefbebd95b5
Step 2/3 : RUN pip install xgboost
 ---> Using cache
 ---> 48e9fa4e03d8
Step 3/3 : RUN python -c "import xgboost as xgb"
 ---> Running in e8ff809ea427
Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!

[hcho3]

@harupy I think I found a fix. See #6480 (comment)

[harupy]

@hcho3 Thanks for the quick action. The new wheel worked. Here's the updated dockerfile.

FROM python:3.7

# download link:
# https://xgboost-examples.s3-us-west-2.amazonaws.com/xgboost-1.3.0-py3-none-manylinux2010_x86_64.whl
COPY xgboost-1.3.0-py3-none-manylinux2010_x86_64.whl .
RUN pip install xgboost-1.3.0-py3-none-manylinux2010_x86_64.whl
RUN python -c "import xgboost"

[hcho3]

The 1.3.0.post0 version is now available on PyPI: https://pypi.org/project/xgboost/1.3.0.post0/#files. It contains the hot fix (#6482) and should not crash upon import.

[harupy]

@hcho3 Thanks you 👍