dm.xmlsec.binding broken? #30

navneil · 2014-10-25T02:03:58Z

I can't get dm.xmlsec.binding to work on my fedora host. It just core dumps. Has anyone gotten this to work. As far as I can tell, I have everything installed cleanly. I find dm.xmlsec.binding to be very unreliable and modified python-saml to use the xmlsec1 binary instead.

pitbulk · 2014-10-27T15:39:55Z

@navneil , Why do you say dm.xmlsec.binding is unreliable?

dm.xmlsec.binding is used in dm.zope.saml2. After review many libraries, I choosedm.xmlsec.binding.
It has documentation https://pypi.python.org/pypi/dm.xmlsec.binding/1.3.1 and is maintained.

What would you propose?

navneil · 2014-10-28T18:49:12Z

I can't get dm.xmlsec.binding to work on different linux distributions. Even after installing cleanly, it core dumps on import dm.xmlsec.binding. What distribution are you using?

Also, have you thought of using the xmlsec1 binaries instead of the python bindings.

pitbulk · 2014-10-28T18:53:44Z

I tested it on Redhat and Ubuntu. I dislike the use of xmlsec1 binaries execution.
How are you installing the python-saml library? Do you use pip?

Can you take a look on this Travis trace? https://travis-ci.org/onelogin/python-saml As you see is installed as expected.

Also, take a look on the installation section of: https://pypi.python.org/pypi/dm.xmlsec.binding/1.3.1

What you get after those commands?

sudo yum install python-setuptools python-setuptools-devel python-pip
pip install python-saml

cjsteinke · 2014-10-28T22:42:56Z

Hi, I found this during a google search and I am having the same problem.
I get a segmentation fault when I try to load the module as well.

This is on RHEL6.6, I followed the instructions from the Travis trace (with modifications for RHEL) and ensured all the necessary packages are installed.

In order to get past the M2Crypto failures I needed to download the package from the python web site and then apply the fedora_setup from the git clone of M2Crypto (which is an older version than the one available from python pypi)because it's NOT included with the download from pypi.

I also needed to install, which isn't mentioned anywhere, xmlsec1-openssl, xmlsec1-openssl-devel, libtool-ltdl and libtool-ltdl-devel (added the -devel packages to be on the safe side) Thi

Here is output from the coverage run

root@gandalf # coverage run --source=src/onelogin/saml2 --rcfile=tests/coverage.rc setup.py test
running test
running egg_info
writing requirements to src/python_saml.egg-info/requires.txt
writing src/python_saml.egg-info/PKG-INFO
writing top-level names to src/python_saml.egg-info/top_level.txt
writing dependency_links to src/python_saml.egg-info/dependency_links.txt
reading manifest file 'src/python_saml.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found under directory '*'
warning: no previously-included files matching '*.py[co]' found under directory '*'
writing manifest file 'src/python_saml.egg-info/SOURCES.txt'
running build_ext
Segmentation fault (core dumped)

Here is my GDB run:

root@gandalf # gdb python2.7
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/rh/python27/root/usr/bin/python2.7...(no debugging symbols found)...done.
Missing separate debuginfos, use: debuginfo-install python27-python-2.7.5-10.el6.x86_64
(gdb) run
Starting program: /opt/rh/python27/root/usr/bin/python2.7 
[Thread debugging using libthread_db enabled]
Python 2.7.5 (default, Dec  3 2013, 08:35:16) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import dm.xmlsec.binding

Program received signal SIGSEGV, Segmentation fault.
__pyx_f_4lxml_5etree_funicode (__pyx_v_s=0x10 <Address 0x10 out of bounds>) at src/lxml/lxml.etree.c:26833
26833   src/lxml/lxml.etree.c: No such file or directory.
    in src/lxml/lxml.etree.c

Any thoughts?

Thanks for any help.

cjsteinke · 2014-10-29T00:22:08Z

For shiz and giggles I tried on Fedora Core 20 and received the same results.

[root@sauron python-saml]#  coverage run --source=src/onelogin/saml2 --rcfile=tests/coverage.rc setup.py test
running test
running egg_info
writing requirements to src/python_saml.egg-info/requires.txt
writing src/python_saml.egg-info/PKG-INFO
writing top-level names to src/python_saml.egg-info/top_level.txt
writing dependency_links to src/python_saml.egg-info/dependency_links.txt
reading manifest file 'src/python_saml.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found under directory '*'
warning: no previously-included files matching '*.py[co]' found under directory '*'
writing manifest file 'src/python_saml.egg-info/SOURCES.txt'
running build_ext
Segmentation fault

pitbulk · 2014-10-29T00:30:47Z

Hi @cjsteinke and @navneil , Sorry to see that you experiencing problems installing python-saml

@cjsteinke , I install M2Crypto using pypi: https://pypi.python.org/pypi/M2Crypto (openssl and swig are dependences from this library).
You can read in the documentation of python-saml, at https://github.com/onelogin/python-saml#dependences you can see libxmlsec1-dev, and libxmlsec1-openssl is a dependece of this package.

I googled the error and it seems a bug with the lxml library (lxml is a dependece of dm.xmlsec.binding): https://bugs.launchpad.net/lxml/+bug/928333

pitbulk · 2014-10-29T00:50:56Z

Hi @navneil and @cjsteinke , sorry to see that python-saml is not working on Fedora.

In python-saml I install M2Crypto using pypi (https://pypi.python.org/pypi/M2Crypto). This library requires openssl and swig.

I googled the error and it seems a bug on the lxml library (a dependence of dm.xmlsec.binding)

pitbulk · 2014-10-29T00:53:13Z

I asked a friend to try get it working on Fedora.

@cjsteinke can you try to use the lxml library? Maybe an issue that this library was not compiled correctly. Did you install the python-dev, libxml2-dev and libxslt-dev libraries?

In Ubuntu, when building 'dm.xmlsec.binding' I see:

    building 'dm.xmlsec.binding._xmlsec' extension
    gcc -pthread -fno-strict-aliasing -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -D__XMLSEC_FUNCTION__=__FUNCTION__ -DXMLSEC_NO_SIZE_T=1 -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_XKMS=1 -DXMLSEC_NO_CRYPTO_DYNAMIC_LOADING=1 -DXMLSEC_OPENSSL_098=1 -DXMLSEC_CRYPTO_OPENSSL=1 -DXMLSEC_CRYPTO=openssl -I/home/travis/virtualenv/python2.7.8/build/dm.xmlsec.binding/lxml-3.4.0-py2.7-linux-x86_64.egg/lxml -I/home/travis/virtualenv/python2.7.8/build/dm.xmlsec.binding/lxml-3.4.0-py2.7-linux-x86_64.egg/lxml/includes -Isrc -I/usr/include/libxml2 -I/usr/include/xmlsec1 -I/opt/python/2.7.8/include/python2.7 -c src/_xmlsec.c -o build/temp.linux-x86_64-2.7/src/_xmlsec.o
    gcc -pthread -shared -L/opt/python/2.7.8/lib -Wl,-rpath=/opt/python/2.7.8/lib build/temp.linux-x86_64-2.7/src/_xmlsec.o -L/usr/lib -L/usr/lib/x86_64-linux-gnu -L/opt/python/2.7.8/lib -lxml2 -lxmlsec1-openssl -lxmlsec1 -lxslt -lssl -lcrypto -lpython2.7 -o build/lib.linux-x86_64-2.7/dm/xmlsec/binding/_xmlsec.so

pitbulk · 2014-10-29T08:44:39Z

Can you try to execute this in order to know the versión installed:

import sys
from lxml import etree

print("%-20s: %s" % ('Python', sys.version_info))
print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))
print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))
print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))
print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))
print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))

Also in the dm.xmlsec.binding documentation appears:

This package interfaces with lxml via its Cython interface (described in etreepublic.pxd).
Some operating system installations for lxml lack the respective files. In those cases,
you may need to download an lxml source distribution and let the environment variable
LXML_HOME point to its root.

don't know if maybe is missing this LXML_HOME path.

Have you tried to install lxml using pypi? https://pypi.python.org/pypi/lxml/3.4.0

navneil · 2014-10-30T01:56:20Z

I got the normal source but I still get dm.xmlsec.binding to crash:

Python 2.7.5 (default, Dec 3 2013, 08:35:16)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import dm.xmlsec.binding

Program received signal SIGSEGV, Segmentation fault.
__pyx_f_4lxml_5etree_funicode (__pyx_v_s=0x10 <Address 0x10 out of bounds>)
at src/lxml/lxml.etree.c:26833
26833 if (!__pyx_t_1) break;

This is what I am using:

Python : sys.version_info(major=2, minor=7, micro=5, releaselevel='final', serial=0)
lxml.etree : (3, 4, 0, 0)
libxml used : (2, 7, 6)
libxml compiled : (2, 7, 6)
libxslt used : (1, 1, 26)
libxslt compiled : (1, 1, 26)

On Wed, Oct 29, 2014 at 1:55 AM, Sixto Martin notifications@github.com
wrote:

In /src/lxml there are only .pyx files, no .c

So you need to get "normal release sources" to have the C files
pre-compiled, e.g. from http://pypi.python.org/pypi/lxml. If you check
out from git, you need Cython.

http://stackoverflow.com/questions/14588404/build-lxml-pre-compiled-c-cython-files-missing-solved

—
Reply to this email directly or view it on GitHub
#30 (comment).

pitbulk · 2014-10-30T14:24:56Z

I sent a mail to the author of dm.xmlsec.binding (Dieter).
He was very kind helping me with some doubts with the library in the past. I hope he has a solution for this problem.

Sorry for the inconveniences.

cjsteinke · 2014-10-31T00:18:06Z

Output:

Python 2.7.5 (default, Dec  3 2013, 08:35:16) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> from lxml import etree
>>> print("%-20s: %s" % ('Python', sys.version_info))
Python              : sys.version_info(major=2, minor=7, micro=5, releaselevel='final', serial=0)
>>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))
lxml.etree          : (3, 4, 0, 0)
>>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))
libxml used         : (2, 7, 6)
>>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))
libxml compiled     : (2, 7, 6)
>>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))
libxslt used        : (1, 1, 26)
>>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))
libxslt compiled    : (1, 1, 26)

I was able to upgrade lxml, I had 2.2.3 installed and it upgraded it to 3.4.0.

I also downloaded lxml and pointed LXML_HOME to it's root and no luck.

Thanks for your help.

pitbulk · 2014-10-31T00:49:04Z

I found this resource:
http://nz-realme.readthedocs.org/en/latest/installation.html#libxml2-and-xmldsig
Can you try install that libxml2 as suggested?

navneil · 2014-10-31T00:51:02Z

I finally got dm.xmlsec.binding to work on Fedora but I needed to comment out this line

transformByHref = dict((t.href, t) for t in transforms)

in file dm/xmlsec/binding/init.py

I don't know what this line does but I am able to run the python-saml toolkit and was able to get the demo-flask app to work.

pitbulk · 2014-10-31T01:01:32Z

Nice, I mailed to Dieter that info. Let's see if he can fix that issue.

Thanks for getting involved!

cjsteinke · 2014-10-31T17:48:39Z

I confirmed that I installed libxml2,but same story.

Not sure if this will help, I ran a GDB session in the lxml-3.4.0 source tree

Contents of script to load dm.xmlsec.binding

cat ../onelogin/python-saml/loadxmlsec.py 
#!/usr/bin/env python2.7
import dm.xmlsec.binding

print "Foo!"

gdb python2.7
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/rh/python27/root/usr/bin/python2.7...(no debugging symbols found)...done.
Missing separate debuginfos, use: debuginfo-install python27-python-2.7.5-10.el6.x86_64
(gdb) run ../onelogin/python-saml/loadxmlsec.py 
Starting program: /opt/rh/python27/root/usr/bin/python2.7 ../onelogin/python-saml/loadxmlsec.py
[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.
__pyx_f_4lxml_5etree_funicode (__pyx_v_s=0x10 <Address 0x10 out of bounds>) at src/lxml/lxml.etree.c:26833
26833       if (!__pyx_t_1) break;
(gdb) bt
#0  __pyx_f_4lxml_5etree_funicode (__pyx_v_s=0x10 <Address 0x10 out of bounds>) at src/lxml/lxml.etree.c:26833
#1  0x00007ffff123da2e in pyunicode (__pyx_v_s=<value optimized out>) at src/lxml/lxml.etree.c:179443
#2  0x00007ffff18061df in __pyx_f_2dm_6xmlsec_7binding_7_xmlsec_xmlChar2py (__pyx_v_xs=<value optimized out>) at src/_xmlsec.c:8352
#3  0x00007ffff18065b1 in __pyx_pf_2dm_6xmlsec_7binding_7_xmlsec_9Transform_4href___get__ (o=<value optimized out>, x=<value optimized out>) at src/_xmlsec.c:2421
#4  __pyx_pw_2dm_6xmlsec_7binding_7_xmlsec_9Transform_4href_1__get__ (o=<value optimized out>, x=<value optimized out>) at src/_xmlsec.c:2399
#5  __pyx_getprop_2dm_6xmlsec_7binding_7_xmlsec_9Transform_href (o=<value optimized out>, x=<value optimized out>) at src/_xmlsec.c:8651
#6  0x00007ffff7c98968 in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#7  0x00007ffff7cbfd1b in _PyObject_GenericGetAttrWithDict () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#8  0x00007ffff7d18948 in PyEval_EvalFrameEx () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#9  0x00007ffff7c9e337 in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#10 0x00007ffff7c7f7fb in PyIter_Next () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#11 0x00007ffff7cbb78f in PyDict_MergeFromSeq2 () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#12 0x00007ffff7cbc0a7 in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#13 0x00007ffff7cd74d8 in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#14 0x00007ffff7c80913 in PyObject_Call () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#15 0x00007ffff7d1c7c6 in PyEval_EvalFrameEx () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#16 0x00007ffff7d1ec4e in PyEval_EvalCodeEx () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#17 0x00007ffff7d1ed62 in PyEval_EvalCode () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#18 0x00007ffff7d2e522 in PyImport_ExecCodeModuleEx () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#19 0x00007ffff7d2e948 in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#20 0x00007ffff7d2f77a in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#21 0x00007ffff7d2fe11 in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#22 0x00007ffff7d30084 in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#23 0x00007ffff7d30f10 in PyImport_ImportModuleLevel () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#24 0x00007ffff7d1681f in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#25 0x00007ffff7c80913 in PyObject_Call () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#26 0x00007ffff7d16d03 in PyEval_CallObjectWithKeywords () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#27 0x00007ffff7d18b16 in PyEval_EvalFrameEx () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#28 0x00007ffff7d1ec4e in PyEval_EvalCodeEx () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#29 0x00007ffff7d1ed62 in PyEval_EvalCode () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#30 0x00007ffff7d3914c in ?? () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#31 0x00007ffff7d39220 in PyRun_FileExFlags () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#32 0x00007ffff7d3a7bf in PyRun_SimpleFileExFlags () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#33 0x00007ffff7d4c2b6 in Py_Main () from /opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
#34 0x0000003df9a1ed5d in __libc_start_main () from /lib64/libc.so.6
#35 0x0000000000400649 in _start ()
(gdb) info threads
* 1 Thread 0x7ffff7bff700 (LWP 1252)  __pyx_f_4lxml_5etree_funicode (__pyx_v_s=0x10 <Address 0x10 out of bounds>) at src/lxml/lxml.etree.c:26833
(gdb)

pitbulk · 2014-11-03T08:30:41Z

Dieter replied:

Looks as if the "libxmlsec1" version in this setup in not fully appropriate -- maybe, the version is too old (as this is the case for Ubuntu 10.04).

"dm.xmlsec.binding" is quite sensible to the correct version of "libxmlsec1".

"libxmlsec1" identifies transforms by so called "transform id"s.
Those ids as in fact pointers to structures with (among others) an "href" field.

In order to make a transform accessible by Python, "dm.xmlsec.binding" must provide a binding for the corresponding "transform id"s.
Especially, it must list all transform ids it wants to support.
I have created this list based on "xmlsec 1.2.18.

The problem you observe seems to indicate that one of the listed transforms is missing (I would expect a linking error in this case)
or its "href" field has an inappropriate value.

pitbulk · 2014-11-03T09:10:43Z

And after check the error trace:

This strenghens my assumption from the previous message.

As you can see, "dm.xmlsec.binding" tries to get the "href" from a transform
("__pyx_pf_2dm_6xmlsec_7binding_7_xmlsec_9Transform_4href___get__").
It gets some value (maybe "NULL", maybe "0x10") and hands it over to the "xmlChar2py" function of "dm.xmlsec.binding". This in turn
calls the "funicode" function from "lxml" which crashes due to a bad pointer ("0x10").

I interpret this as follows:
some of the transforms in the transform list used by "dm.xmlsec.binding" are not defined by the local "libxmlsec1" C library (version probably too old).
Rather than generating an error during the dynamic link phase, the linker returns an inappropriate value for the missing symbols
(likely "NULL"). Determining the "href" may lead to "0x10" which lets "lxml"'s "funicode" crash.


"dm.xmlsec.binding" generates its transform list and it might be possible to filter out transforms with inappropriate ids (e.g. "NULL" ids).
However, a too old version of "libxmlsec1" may have other consequences besides unsupported transforms. E.g. the "libxmlsec1" version
of Ubuntu 10.04 fails to support a function vital to "dm.xmlsec.binding", a function vital for "lxml" integration. Working to avoid the SIGSEGV
with the transforms would invariably cause another SIGSEGV later.
In this case, nothing would help other than an upgrade of "libxmlsec1".

cjsteinke · 2014-11-03T17:39:26Z

These are the version of xmlsec1 that have installed on my system:

xmlsec1-openssl-devel-1.2.20-4.el6.x86_64
xmlsec1-devel-1.2.20-4.el6.x86_64
xmlsec1-openssl-1.2.20-4.el6.x86_64
xmlsec1-1.2.20-4.el6.x86_64

So these version are newer than the 1.2.18 version used by Dieter.

I have noticed that packages from Ubuntu sometimes have bug fixes and enhancements that don't get pushed out to or included in other releases of the package for other distros and it's really annoying.
In one particular case I was working with building a VPN server using PPTP and the Ubuntu version worked fine with IOS 6 using the same verison, off only by a subversion, but on RHEL or CENTOS it failed to work with IOS 6 devices, because there was a bug that was fixed and applied to the Ubuntu package but was not available in RHEL or CENTOS. Not even Debian had the fix in it's package, of the same 'major' version and the equivalent subversion was not available on any distro, with the exception of Ubuntu.

I hope this is not the case as I work in a RedHat shop.

Are you running 1.2.18 or 1.2.20 or a later version?

Thanks!

pitbulk · 2014-11-11T16:36:47Z

I'm using ubuntu 13.10 with xmlsec1 1.2.18 (openssl) where the python-toolkit is working.
dm.xmlsec.binding defines those transformations:

TransformInclC14N, TransformInclC14NWithComments, TransformInclC14N11,
TransformInclC14N11WithComments, TransformExclC14N, TransformExclC14NWithComments,
TransformEnveloped, TransformXPath, TransformXPath2, TransformXPointer, TransformXslt,
TransformRemoveXmlTagsC14N, TransformVisa3DHack, TransformAes128Cbc,
TransformAes192Cbc, TransformAes256Cbc, TransformKWAes128, TransformKWAes192, 
TransformKWAes256, TransformDes3Cbc, TransformKWDes3, TransformDsaSha1,
TransformHmacMd5, TransformHmacRipemd160, TransformHmacSha1, 
TransformHmacSha224,TransformHmacSha256, TransformHmacSha384, 
TransformHmacSha512, TransformMd5, TransformRipemd160, TransformRsaMd5, 
TransformRsaRipemd160, TransformRsaSha1, TransformRsaSha224, TransformRsaSha256, 
TransformRsaSha384, TransformRsaSha512, TransformRsaPkcs1, TransformRsaOaep, 
TransformSha1, TransformSha224, TransformSha256, TransformSha384, TransformSha512

I think we need to compare the /usr/include/xmlsec1/xmlsec/transforms.h from the ubuntu version from the 1.2.18 with the one that is installed using rhel/centOS to see what transform does not exists in the rhel/centOS branch.

This is the ubuntu version:
https://gist.github.com/pitbulk/ab258762a9487dc7030e
that is the same that the lastest xmlsec version: https://www.aleksey.com/xmlsec/download.html

pitbulk · 2015-01-14T13:55:14Z

@navneil and @cjsteinke There is a new release of the dm.xmlsec.binding library.

Can you please check if this new release solve your problems?

navneil · 2015-01-16T02:58:11Z

I gave it a try and it sill doesn't work :(

FxChiP · 2015-01-23T23:53:49Z

Can confirm the transforms.h given in the gist matches the transforms.h on CentOS 6 save a trailing newline, still experiencing this issue.

FxChiP · 2015-01-24T00:00:47Z

Can confirm https://bugs.launchpad.net/lxml/+bug/928333 is unrelated in my case

FxChiP · 2015-01-24T01:03:20Z

All transforms listed are present in the various xmlsec1 header files on CentOS 6 as well.

FxChiP · 2015-01-24T01:13:38Z

There are two transforms referenced by the binding that do have NULLs in the xmlsec1 library: TransformRemoveXmlTagsC14N and TransformVisa3DHack. However, xmlChar2py will return None on NULL pointers.

FxChiP · 2015-01-24T01:45:06Z

Simply importing the shared object is enough:

>>> import dm.xmlsec.binding._xmlsec

Program received signal SIGSEGV, Segmentation fault.
__pyx_f_4lxml_5etree_funicode (__pyx_v_s=0x10 <Address 0x10 out of bounds>)
    at src/lxml/lxml.etree.c:26753
26753   src/lxml/lxml.etree.c: No such file or directory.
    in src/lxml/lxml.etree.c
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6_5.4.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libgcrypt-1.4.5-11.el6_4.x86_64 libgpg-error-1.7-4.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libtool-ltdl-2.2.6-15.5.el6.x86_64 libxml2-2.7.6-17.el6_6.1.x86_64 libxslt-1.1.26-2.el6_3.1.x86_64 openssl-1.0.1e-30.el6_6.5.x86_64 xmlsec1-1.2.20-4.el6.x86_64 xmlsec1-openssl-1.2.20-4.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  __pyx_f_4lxml_5etree_funicode (__pyx_v_s=0x10 <Address 0x10 out of bounds>)
    at src/lxml/lxml.etree.c:26753
#1  0x00007fffef8a55fe in pyunicode (__pyx_v_s=<value optimized out>)
    at src/lxml/lxml.etree.c:179019
#2  0x00007ffff1e0dcc7 in __pyx_f_2dm_6xmlsec_7binding_7_xmlsec_xmlChar2py (
    o=<value optimized out>, x=<value optimized out>) at src/_xmlsec.c:8352
#3  __pyx_pf_2dm_6xmlsec_7binding_7_xmlsec_9Transform_4href___get__ (
    o=<value optimized out>, x=<value optimized out>) at src/_xmlsec.c:2421
#4  __pyx_pw_2dm_6xmlsec_7binding_7_xmlsec_9Transform_4href_1__get__ (
    o=<value optimized out>, x=<value optimized out>) at src/_xmlsec.c:2399
#5  __pyx_getprop_2dm_6xmlsec_7binding_7_xmlsec_9Transform_href (
    o=<value optimized out>, x=<value optimized out>) at src/_xmlsec.c:8651
#6  0x0000000000451d12 in _PyObject_GenericGetAttrWithDict (
    obj=0x7ffff7f70ac8, name=<value optimized out>) at Objects/object.c:1434
#7  PyObject_GenericGetAttr (obj=0x7ffff7f70ac8, name=<value optimized out>)
    at Objects/object.c:1456
#8  0x00000000004a4a37 in PyEval_EvalFrameEx (f=<value optimized out>, 
    throwflag=<value optimized out>) at Python/ceval.c:2256
#9  0x000000000050885d in gen_send_ex (gen=0x7fffee734550, arg=0x0, 
    exc=<value optimized out>) at Objects/genobject.c:84
#10 0x00000000004188cb in PyIter_Next (iter=<value optimized out>)
    at Objects/abstract.c:3107
...

ThemGit · 2017-03-06T13:56:21Z

Hello @pitbulk ,

i am running on Red Hat Enterprise Linux Server release 6.8

i have followed all the steps mentioned above and installed

libxml2-devel xmlsec1-devel python-devel xmlsec1-openssl-devel libtool-ltdl-devel

running pip install python-saml completes fine

i change -DXMLSEC_NO_SIZE_T

but then i still get Segmentation fault (core dumped)

Are there any other suggestions on how to overcome this? It seems quite strange that after all this time it is still an issue.

Update
by manually compiling xmlsec1-devel my issue was resolved.

I followed the directions here:

skortchmark9 · 2017-06-15T19:28:09Z

We're using this library for SSO and are pretty unhappy with how difficult it is to install. All of the problems come from dm.xmlsec.binding. The recent 1.3.3 upgrade broke us again. We're considering switching to: https://github.com/rohe/pysaml2 as an alternative.@navneil is your fork which uses xmlsec1 available anywhere?

pitbulk · 2017-06-15T19:30:28Z

@skortchmark9

You can use python3-saml that uses another xmlsec lib...not dm.xmlsec.binding

taldev-no · 2017-06-15T19:36:29Z

@skortchmark9 I'd just like to echo what @pitbulk said, switching to python3-saml (works on 2.7 btw, we're still on 2.7.11) saved us a lot of setup and configuration hassle (also the API is better designed and easier to use IMO).

ThemGit · 2017-06-27T08:27:48Z

@Heilage Did you need to do any changes in the code that uses python-saml, or are the method calls etc. exactly the same on python3-saml?

taldev-no · 2017-06-27T09:25:21Z

@ThemGit It's been a while since we did the switch, but I seem to remember not having to make many changes, maybe a few new imports and stuff. Unless I'm very much mistaken, it was a very easy switch.

ThemGit · 2017-06-27T09:26:48Z

@Heilage Thanks! I will post back then as well once i make the switch on our own code.

ender672 · 2017-09-14T17:32:03Z

As an alternative to editing /usr/bin/xmlsec1-config, you can set the environment variable CFLAGS=-DXMLSEC_NO_SIZE_T.

For example:

$ pip uninstall dm.xmlsec.binding
$ CFLAGS=-DXMLSEC_NO_SIZE_T pip install --no-cache-dir dm.xmlsec.binding

Terhands · 2017-10-02T21:57:02Z

Just a heads up, there is a similar issue on MacOS 10.12 which prevents the successful installation of dm.xmlsec.binding.

Here is the apple thread with more information: https://forums.developer.apple.com/thread/87829

Emerson · 2017-10-20T17:03:53Z

Dealing with an issue here as well on Ubuntu 14.04 with all required packages installed (I think). If anyone has any knowledge of what could be wrong I'd really appreciate it, or likewise, if I figure out the issue I'll respond in this thread. Here is the error I see when trying to install v2.3.3:

(virtualenv) deploy@vagrant-ubuntu-trusty-64:/srv/git$ pip install python-saml
Collecting python-saml
  Using cached python_saml-2.3.0-py2-none-any.whl
Collecting dm.xmlsec.binding==1.3.3 (from python-saml)
  Using cached dm.xmlsec.binding-1.3.3.tar.gz
    Complete output from command python setup.py egg_info:
    x86_64-linux-gnu-gcc: internal compiler error: Killed (program cc1)
    Please submit a full bug report,
    with preprocessed source if appropriate.
    See <file:///usr/share/doc/gcc-4.8/README.Bugs> for instructions.
    Building lxml version 4.1.0.
    Building without Cython.
    Using build configuration of libxslt 1.1.28
    Compile failed: command 'x86_64-linux-gnu-gcc' failed with exit status 4
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-6ZN_Bm/dm.xmlsec.binding/setup.py", line 224, in <module>
        **setupArgs
      File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "/srv/virtualenv/local/lib/python2.7/site-packages/setuptools/dist.py", line 335, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "/srv/virtualenv/local/lib/python2.7/site-packages/setuptools/dist.py", line 456, in fetch_build_eggs
        replace_conflicting=True,
      File "/srv/virtualenv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 863, in resolve
        replace_conflicting=replace_conflicting
      File "/srv/virtualenv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1141, in best_match
        return self.obtain(req, installer)
      File "/srv/virtualenv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1153, in obtain
        return installer(requirement)
      File "/srv/virtualenv/local/lib/python2.7/site-packages/setuptools/dist.py", line 522, in fetch_build_egg
        return cmd.easy_install(req)
      File "/srv/virtualenv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 672, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/srv/virtualenv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 698, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/srv/virtualenv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 879, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/srv/virtualenv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1118, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/srv/virtualenv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1106, in run_setup
        raise DistutilsError("Setup script exited with %s" % (v.args[0],))
    distutils.errors.DistutilsError: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 4
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-6ZN_Bm/dm.xmlsec.binding/

Emerson · 2017-10-23T17:58:57Z

I wanted to followup on my previous comment. It turns out the issue was related to memory issues. I went ahead and created a 1GB swap on my box and pip install works fine now. Hope this helps someone else ✌️

juliedavila · 2017-10-26T12:02:03Z

Also confirming CentOS7 is fixed by adding the -DXMLSEC_NO_SIZE_T flag and reinstall dm.xmlsec.binding with the no cache option.

Mirrors reports by @Heilage and @repagh

jerkyrs · 2017-10-26T13:01:52Z

This is referenced in another thread #177

I did the following within the venv on Centos 7
export ARCHFLAGS='-arch x86_64'
pip install pysaml2==3.0.0

I also ran the above xmlsec configuration change

It also resulted in deadlock on uwsgi that resulted with the following error
ImportError: lxml.etree does not export expected C function adoptExternalDocument

I will need to rebuild via kickstart/puppet to verify successful build, but this was also causing issues on our Centos6 builds.

mathieu-lacage · 2018-05-04T14:30:04Z

is there a reason dm.xmlsec.binding is still using xmlsec1-config instead of using pkg-config --cflags ?

i.e., borg's patch to xmlsec1-config to add -DXMLSEC_NO_SIZE_T works.

Note, this is on stock FC27

python-saml depends on dm.xmlsec.binding which is a python binding of xmlsec C library. but without -DXMLSEC_NO_SIZE_T the compiled dm.xmlsec.binding segfaults when `import dm.xmlsec.binding`. see SAML-Toolkits/python-saml#30 and 4teamwork/ftw.saml2auth#3 . in long term, we might want to switch to python-saml, see http://tracker.ceph.com/issues/37081 Signed-off-by: Kefu Chai <kchai@redhat.com>

xanterx · 2019-05-15T08:59:33Z

As an alternative to editing /usr/bin/xmlsec1-config, you can set the environment variable CFLAGS=-DXMLSEC_NO_SIZE_T.

For example:
$ pip uninstall dm.xmlsec.binding
$ CFLAGS=-DXMLSEC_NO_SIZE_T pip install --no-cache-dir dm.xmlsec.binding

For Pipenv I used
pipenv run pip uninstall dm.xmlsec.binding
CFLAGS=-DXMLSEC_NO_SIZE_T pipenv run pip install --no-cache-dir dm.xmlsec.binding

pitbulk closed this as completed Oct 29, 2014

pitbulk reopened this Oct 29, 2014

pitbulk mentioned this issue Oct 31, 2014

Direction to take for XML digital signature generation/verification darrylcousins/nzrealme#1

Closed

pitbulk mentioned this issue May 9, 2017

Failure to handle "saml2" and "saml2p" xml namespaces #189

Closed

piotch mentioned this issue May 31, 2017

Upgrade to version 1.3.3 of dm.xmlsec.binding #199

Merged

pitbulk mentioned this issue Jun 12, 2017

Got different errors on different OS, please help #200

Closed

fstuck37 mentioned this issue Aug 2, 2017

OneLogin SAML Instructions Netflix/security_monkey#779

Closed

jerkyrs mentioned this issue Oct 26, 2017

OSX libxml2 is not compatible with lxml in venv assembl/assembl#128

Closed

thomasDOTwtf mentioned this issue Jan 22, 2018

Segmentation fault when run create_db.py thomasDOTwtf/PowerDNS-Admin#19

Closed

AlanCoding mentioned this issue Mar 2, 2018

Update monkey patch for python-saml ansible/awx#1421

Closed

jerkyrs mentioned this issue Jun 5, 2018

Signature validation failed. Everything seems right, though. #166

Closed

pitbulk mentioned this issue Aug 26, 2018

Error when running pip install python-saml #237

Closed

tchaikov mentioned this issue Nov 16, 2018

mgr/dashboard: add pytest plugin: faulthandler ceph/ceph#25053

Merged

3 tasks

tchaikov mentioned this issue Nov 16, 2018

test/dashboard: fix segfault when importing dm.xmlsec.binding ceph/ceph#25139

Merged

3 tasks

pitbulk mentioned this issue Nov 24, 2020

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. #276

Closed

pitbulk added question Infornative labels Dec 26, 2022

dm.xmlsec.binding broken? #30

dm.xmlsec.binding broken? #30

Comments

navneil commented Oct 25, 2014

pitbulk commented Oct 27, 2014

navneil commented Oct 28, 2014

pitbulk commented Oct 28, 2014

cjsteinke commented Oct 28, 2014

cjsteinke commented Oct 29, 2014

pitbulk commented Oct 29, 2014

pitbulk commented Oct 29, 2014

pitbulk commented Oct 29, 2014

pitbulk commented Oct 29, 2014

navneil commented Oct 30, 2014

pitbulk commented Oct 30, 2014

cjsteinke commented Oct 31, 2014

pitbulk commented Oct 31, 2014

navneil commented Oct 31, 2014

pitbulk commented Oct 31, 2014

cjsteinke commented Oct 31, 2014

pitbulk commented Nov 3, 2014

pitbulk commented Nov 3, 2014

cjsteinke commented Nov 3, 2014

pitbulk commented Nov 11, 2014

pitbulk commented Jan 14, 2015

navneil commented Jan 16, 2015

FxChiP commented Jan 23, 2015

FxChiP commented Jan 24, 2015

FxChiP commented Jan 24, 2015

FxChiP commented Jan 24, 2015

FxChiP commented Jan 24, 2015

ThemGit commented Mar 6, 2017 • edited

skortchmark9 commented Jun 15, 2017

pitbulk commented Jun 15, 2017

taldev-no commented Jun 15, 2017

ThemGit commented Jun 27, 2017

taldev-no commented Jun 27, 2017

ThemGit commented Jun 27, 2017

ender672 commented Sep 14, 2017

Terhands commented Oct 2, 2017

Emerson commented Oct 20, 2017

Emerson commented Oct 23, 2017

juliedavila commented Oct 26, 2017

jerkyrs commented Oct 26, 2017

mathieu-lacage commented May 4, 2018

xanterx commented May 15, 2019 • edited

ThemGit commented Mar 6, 2017 •

edited

xanterx commented May 15, 2019 •

edited