PY_SSIZE_T_CLEAN macro must be defined for '#' formats

[paulefoa]

Describe the bug
I got this error when trying to produce the message using aiokafka 0.8.1 and Python 3.11.6

This urls might useful for someone who is smarter than me
https://bugs.python.org/issue40943
https://peps.python.org/pep-0353/

Here's the stack trace:

[Tue 2023-10-10 11:13:34+0000] ERROR [aiokafka.producer.sender._sender_routine:167] Unexpected error in sender routine
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 155, in _sender_routine
    task.result()
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 260, in _send_produce_req
    await handler.do(node_id)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 714, in do
    request = self.create_request()
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 678, in create_request
    (tp.partition, batch.get_data_buffer())
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/message_accumulator.py", line 234, in get_data_buffer
    return self._builder._build()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/message_accumulator.py", line 87, in _build
    self._buffer = self._builder.build()
                   ^^^^^^^^^^^^^^^^^^^^^
  File "aiokafka/record/_crecords/default_records.pyx", line 716, in aiokafka.record._crecords.default_records.DefaultRecordBatchBuilder.build
  File "aiokafka/record/_crecords/default_records.pyx", line 701, in aiokafka.record._crecords.default_records.DefaultRecordBatchBuilder._maybe_compress
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
[Tue 2023-10-10 11:13:34+0000] ERROR [asyncio.default_exception_handler:1771] Exception in callback Sender._fail_all(<Task finishe...ch delivery')>)
handle: <Handle Sender._fail_all(<Task finishe...ch delivery')>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 70, in _fail_all
    self._txn_manager.fatal_error(task_exception)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/transaction_manager.py", line 163, in fatal_error
    if self._transaction_waiter.done():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'done'

I've tried to add this lines of code:

#ifndef PY_SSIZE_T_CLEAN
#define PY_SSIZE_T_CLEAN

to aiokafka/record/_crecords/crc32c.h and aiokafka/record/_crecords/crc32c.c
And it worked, but when I used lz4 compression I got this error:

[Tue 2023-10-10 11:13:34+0000] ERROR [aiokafka.producer.sender._sender_routine:167] Unexpected error in sender routine
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 155, in _sender_routine
    task.result()
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 260, in _send_produce_req
    await handler.do(node_id)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 714, in do
    request = self.create_request()
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 678, in create_request
    (tp.partition, batch.get_data_buffer())
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/message_accumulator.py", line 234, in get_data_buffer
    return self._builder._build()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/message_accumulator.py", line 87, in _build
    self._buffer = self._builder.build()
                   ^^^^^^^^^^^^^^^^^^^^^
  File "aiokafka/record/_crecords/default_records.pyx", line 716, in aiokafka.record._crecords.default_records.DefaultRecordBatchBuilder.build
  File "aiokafka/record/_crecords/default_records.pyx", line 701, in aiokafka.record._crecords.default_records.DefaultRecordBatchBuilder._maybe_compress
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
[Tue 2023-10-10 11:13:34+0000] ERROR [asyncio.default_exception_handler:1771] Exception in callback Sender._fail_all(<Task finishe...ch delivery')>)
handle: <Handle Sender._fail_all(<Task finishe...ch delivery')>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/sender.py", line 70, in _fail_all
    self._txn_manager.fatal_error(task_exception)
  File "/usr/local/lib/python3.11/site-packages/aiokafka/producer/transaction_manager.py", line 163, in fatal_error
    if self._transaction_waiter.done():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'done'

So I guess that helps somewhat but kafka.codecs also has this problem?

Environment (please complete the following information):

• aiokafka version: 0.8.1
• kafka-python version: 2.0.2
• Kafka Broker version (kafka-topics.sh --version):

[ods]

Why do you suspect crc32c.c/crc32s.h? They don't include Python.h and don't deal with Python objects conversion.

Do you install binary package or compile it from source? If later, what Cython version do you use? What OS? CI pipeline should have catch the error, if it was in pre-build packages.