Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cyrus-sasl broken in 2023Q4, 2022Q4, 2021Q4 #377

Open
xmerlin opened this issue Apr 9, 2024 · 8 comments
Open

cyrus-sasl broken in 2023Q4, 2022Q4, 2021Q4 #377

xmerlin opened this issue Apr 9, 2024 · 8 comments

Comments

@xmerlin
Copy link

xmerlin commented Apr 9, 2024

After the 2020Q4 release, it is impossible to use the SQL / auxprop plugin of Cyrus-SASL. The bug affects all daemons using Cyrus-SASL for authentication. Specifically, attached are the steps to reproduce the problem on a minimal configuration of Postfix.

install packages postfix postfix-mysql cyrus-sasl cy2-login cy2-plain cy2-sql

CREATE DATABASE

CREATE DATABASE postfix_db;
GRANT SELECT ON postfix_db.* TO 'postfix_user'@'localhost' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
USE postfix_db;

CREATE TABLE virtual_users (
id int(11) NOT NULL auto_increment,
email varchar(100) NOT NULL,
password varchar(100) NOT NULL,
PRIMARY KEY (id),
UNIQUE KEY email (email)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

INSERT INTO virtual_users (email, password) VALUES ('test@example.com', 'password');

configure postfix

/opt/local/etc/postfix/main.cf

smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain = $myhostname
smtpd_recipient_restrictions = permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination
smtpd_sasl_type = cyrus
smtpd_sasl_path = smtpd

broken_sasl_auth_clients = yes

virtual_mailbox_maps = proxy:mysql:/opt/local/etc/postfix/mysql_virtual_mailbox_maps.cf

/opt/local/etc/postfix/mysql_virtual_mailbox_maps.cf

user = postfix_user
password = password
hosts = localhost
dbname = postfix_db
query = SELECT 1 FROM virtual_users WHERE email='%s'

/opt/local/etc/postfix/smtpd.conf

pwcheck_method: auxprop
auxprop_plugin: sql
mech_list: plain login

sql_engine: mysql
sql_hostnames: localhost
sql_user: postfix_user
sql_passwd: password
sql_database: postfix_db
sql_select: SELECT password FROM virtual_users WHERE email = '%u@%r' ;

sql_usessl: 0
sql_verbose: yes
sql_log_level: 7

@jperkin
Copy link
Collaborator

jperkin commented Apr 11, 2024

Some initial notes:

  • The smtpd.conf location needs to be /opt/local/etc/sasl2/smtpd.conf.
  • With that in place, the smtpd process crashes.

The stack is as follows:

core 'core.smtpd.8477' of 8477:	smtpd -n smtp -t inet -u -s 2
 fffffc7fbc7ea343 mysql_cset_escape_slashes (0, 4b6770, 5b0db8, 4) + 83
 fffffc7fbc764125 _mysql_escape_str (4b6770, 5b0db8) + 25
 fffffc7fbc763882 sql_auxprop_lookup (599b50, 48e950, 0, 5c0db1, 10) + 1a2
 fffffc7fbcc9ab33 _sasl_auxprop_lookup (48e950, 0, 5c0db1, 10) + 1a3
 fffffc7fbcc9bacc _sasl_canon_user_lookup (5c0010, 5ac910, 10, 7, 5c0880) + 10c
 fffffc7fbc782676 login_server_mech_step (5b1940, 48e950, 5b2de0, 8, fffffc7fffdff060, fffffc7fffdff05c, 5c0880) + 1a6
 fffffc7fbcca7a89 sasl_server_step (5c0010, 5b2de0, 8, fffffc7fffdff060, fffffc7fffdff05c) + a9
 0000000000429b80 xsasl_cyrus_server_next (5a7b10, 4f67c0, 599500) + a0
 0000000000426bb4 smtpd_sasl_authenticate (fffffc7fffdff1e0, 5b1b30, 0) + 54
 0000000000413320 smtpd_sasl_auth_cmd_wrapper (fffffc7fffdff1e0, 2, 492660) + 50
 00000000004190e2 smtpd_proto (fffffc7fffdff1e0) + 992
 00000000004197e3 smtpd_service (598ca0, fffffc7fffdfff59, fffffc7fffdffda8) + 163
 fffffc7fbc85492c single_server_wakeup (17, 0) + cc
 fffffc7fbcaaaf98 event_loop (ffffffff) + 298
 fffffc7fbc855af8 single_server_main (8, fffffc7fffdffd68, 419680) + d98
 000000000041a617 ???????? ()
 00000000004114d7 _start_crt () + 87
 0000000000411438 _start () + 18

Looking at the source for mysql_cset_escape_slashes():

size_t mysql_cset_escape_slashes(const MARIADB_CHARSET_INFO * cset, char *newstr,
                     const char * escapestr, size_t escapestr_len )
{
  const char   *newstr_s = newstr;
  const char   *newstr_e = newstr + 2 * escapestr_len;
  const char   *end = escapestr + escapestr_len;
  my_bool  escape_overflow = FALSE;

  for (;escapestr < end; escapestr++) {
    char esc = '\0';
    unsigned int len = 0;

    /* check unicode characters */
    if (cset->char_maxlen > 1 && (len = cset->mb_valid(escapestr, end))) {
...

we can see why it's crashing, the function is being called with cset set to NULL, but there is no check in the function for it being valid and it immediately tries to deference it with cset->char_maxlen.

It's likely setting a valid charset will resolve the problem. An open question is why this has changed, and maybe we're no longer specifying a default or something.

@jperkin
Copy link
Collaborator

jperkin commented Apr 11, 2024

Ok, I've found the problem. We have an historical setting in our builds of:

include/pkgoptions.mk:MYSQL_CHARSET=                    utf8

However, this is no longer a valid character set:

MariaDB [(none)]> show character set where charset like '%utf8%';
+---------+---------------+--------------------+--------+
| Charset | Description   | Default collation  | Maxlen |
+---------+---------------+--------------------+--------+
| utf8mb3 | UTF-8 Unicode | utf8mb3_general_ci |      3 |
| utf8mb4 | UTF-8 Unicode | utf8mb4_general_ci |      4 |
+---------+---------------+--------------------+--------+
2 rows in set (0.001 sec)

I'll get that setting removed so that we use the default, which is set as:

databases/mariadb106-client/Makefile.common:MARIADB_CHARSET?=   ${MYSQL_CHARSET:Uutf8mb4}
databases/mariadb106-client/Makefile.common:CMAKE_ARGS+=                -DDEFAULT_CHARSET=${MARIADB_CHARSET}

@xmerlin
Copy link
Author

xmerlin commented Apr 13, 2024

utf8mb4 is related to MySQL 8.x. I have tested everything using MySQL 5.7 and found that the bug lies elsewhere. The SQL plugin was not loaded in Q4 for the years 2023, 2022, and 2021, and no queries were executed on the database (I have already debugged every query in the database).

@xmerlin
Copy link
Author

xmerlin commented Apr 13, 2024

I've tried also with /opt/local/etc/sasl2/smtpd.conf same results

@jperkin
Copy link
Collaborator

jperkin commented Apr 15, 2024

With the change to the default charset, and using all of the configuration files you've provided above, it now works for me:

220 135d7c87-8fb4-4f4e-8e42-dd45f2661dc9.localdomain ESMTP
ehlo localhost
250-135d7c87-8fb4-4f4e-8e42-dd45f2661dc9.localdomain
250-PIPELINING
250-SIZE 51200000
250-VRFY
250-ETRN
250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-DSN
250 CHUNKING
auth login
334 VXNlcm5hbWU6
dGVzdEBleGFtcGxlLmNvbQ==
334 UGFzc3dvcmQ6
cGFzc3dvcmQ=
235 2.7.0 Authentication successful

One issue is that the cy2-sql package is not multi-mysql aware, so it will always be built against whatever the default MySQL implementation is for the branch, so in the case of 2023Q4 that is mariadb 10.6. I'll look at fixing this, so that I can also test against MySQL 5.7.

In the meantime please provide details of how the build against MySQL 5.7 is failing for you. Thanks.

@jperkin
Copy link
Collaborator

jperkin commented Apr 15, 2024

FWIW I just built custom versions of cy2-sql and postfix-mysql that explicitly link against mysql-client-5.7, as well as rebuilding mysql 5.7 with the charset fix, and that is able to authenticate successfully too.

@xmerlin
Copy link
Author

xmerlin commented Apr 15, 2024

I confirm that the charset fix works. It would probably be advisable to modify cy2-sql to make it multi-MySQL aware, so that people can install postfix-mysql/cy2-sql and Percona MySQL on the same machine. As soon as I find five minutes, I will investigate whether the fix also resolves the MySQL client crash in the Percona cluster.

thank you

@xmerlin
Copy link
Author

xmerlin commented Apr 23, 2024

I just tested MySQL Percona 8.0.36-28 and can confirm that after changing the encoding, the MySQL client no longer crashes.

jperkin pushed a commit that referenced this issue May 1, 2024
 - Added --locked to Install Instruction & Update dependencies by @AmmarAbouZor in #376
 - Fixed: Specify arm target on mac build and release by @AmmarAbouZor in #377
 - Update all dependencies to their latest versions
jperkin pushed a commit that referenced this issue May 17, 2024
0.21.0 (2024-03-10)
-------------------

- Improve documentation. [#483]

- Add a minimum version requirement for ``asdf-wcs-schemas``. [#491]

- Fix ``WCS.__str__`` for instances without transforms. [#489]

0.20.0 (2023-11-29)
-------------------

- Replace ``pkg_resources`` with ``importlib.metadata``. [#478]

- Serialize and deserialize ``pixel_shape`` with asdf. [#480]

0.19.0 (2023-09-15)
-------------------

Bug Fixes
^^^^^^^^^

- Synchronize ``array_shape`` and ``pixel_shape`` attributes of WCS
  objects. [#439]

- Fix failures and warnings with numpy 2.0. [#472]

other
^^^^^

- Remove deprecated old ``bounding_box``. The new implementation is released with
  astropy v 5.3. [#458]

- Refactor ``CoordinateFrame.axis_physical_types``. [#459]

- ``StokesFrame`` uses now ``astropy.coordinates.StokesCoord``. [#452]

- Dropped support for Python 3.8. [#451]

- Fixed a call to ``astropy.coordinates`` in ``wcstools.wcs_from_points``. [#448]

- Code and docstrings clean up. [#460]

- Register all available asdf extension manifests from ``asdf-wcs-schemas``
  except 1.0.0 (which contains duplicate tag versions). [#469]

- Register empty extension for 1.0.0 to avoid warning about a missing
  extension when opening old files. [#475]


0.18.3 (2022-12-23)
-------------------
Bug Fixes
^^^^^^^^^

- Fixed a bug in the estimate of pixel scale in the iterative inverse
  code. [#423]

- Fixed constant term in the polynomial used for SIP fitting.
  Improved stability and accuracy of the SIP fitting code. [#427]


0.18.2 (2022-09-07)
-------------------
Bug Fixes
^^^^^^^^^

- Corrected the reported requested forward SIP accuracy and reported fit
  residuals by ``to_fits_sip()`` and ``to_fits()``. [#413, #419]

- Fixed a bug due to which the check for divergence in ``_fit_2D_poly()`` and
  hence in ``to_fits()`` and ``to_fits_sip()`` was ignored. [#414]

New Features
^^^^^^^^^^^^

0.18.1 (2022-03-15)
-------------------
Bug Fixes
^^^^^^^^^

- Remove references to the ``six`` package. [#402]

0.18.0 (2021-12-22)
-------------------
Bug Fixes
^^^^^^^^^

- Updated code in ``region.py`` with latest improvements and bug fixes
  from ``stsci.skypac.regions.py`` [#382]

- Added support to ``_compute_lon_pole()`` for computation of ``lonpole``
  for all projections from ``astropy.modeling.projections``. This also
  extends support for different projections in ``wcs_from_fiducial()``. [#389]

New Features
^^^^^^^^^^^^

- Enabled ``CompoundBoundingBox`` support for wcs. [#375]

- Moved schemas to standalone package ``asdf-wcs-schemas``.
  Reworked the serialization code to use ASDF converters. [#388]

0.17.1 (2021-11-27)
-------------------

Bug Fixes
^^^^^^^^^

- Fixed a bug with StokesProfile and array types. [#384]


0.17.0 (2021-11-17)
-------------------
Bug Fixes
^^^^^^^^^

- `world_axis_object_components` and `world_axis_object_classes` now ensure
  unique keys in `CompositeFrame` and `CoordinateFrame`. [#356]

- Fix issue where RuntimeWarning is raised when there are NaNs in coordinates
  in angle wrapping code [#367]

- Fix deprecation warning when wcs is initialized with a pipeline [#368]

- Use ``CD`` formalism in ``WCS.to_fits_sip()``. [#380]


New Features
^^^^^^^^^^^^
- ``wcs_from_points`` now includes fitting for the inverse transform. [#349]

- Generalized ``WCS.to_fits_sip`` to be able to create a 2D celestial FITS WCS
  from celestial subspace of the ``WCS``. Also, now `WCS.to_fits_sip``
  supports arbitrary order of output axes. [#357]


API Changes
^^^^^^^^^^^
- Modified interface to ``wcs_from_points`` function to better match analogous function
  in astropy. [#349]

- ``Model._BoundingBox`` was renamed to ``Model.ModelBoundingBox``. [#376, #377]

0.16.1 (2020-12-20)
-------------------
Bug Fixes
^^^^^^^^^
- Fix a regression with ``pixel_to_world`` for output frames with one axis. [#342]

0.16.0 (2020-12-18)
-------------------
New Features
^^^^^^^^^^^^

- Added an option to `to_fits_sip()` to be able to specify the reference
  point (``crpix``) of the FITS WCS. [#337]

- Added support for providing custom range of degrees in ``to_fits_sip``. [#339]

Bug Fixes
^^^^^^^^^

- ``bounding_box`` now works with tuple of ``Quantities``. [#331]

- Fix a formula for estimating ``crpix`` in ``to_fits_sip()`` so that ``crpix``
  is near the center of the bounding box. [#337]

- Allow sub-pixel sampling of the WCS model when computing SIP approximation in
  ``to_fits_sip()``. [#338]

- Fixed a bug in ``to_fits_sip`` due to which ``inv_degree`` was ignored. [#339]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants