Fix build

[thijsc]

Attempt to fix the build. Also upgrades to librdkafka 1.2.2.

All individual tests are green based on some random sampling. It just loses connection to the broker after a while. Re-running failing specs immediately after produced a green result.

Not sure why this is breaking exactly.

[thijsc]

I think this might a a docker compose issue, no clue why it randomly stalls. We never see this happen when running this gem in production. Any ideas are welcome.

[rgo]

Yesterday, I was doing some tests and now consumer behaviour have changed.
In Consumer context "#pause and #resume" consumer doesn't get assignment (line 65 fails).

If it helps, I modified wait_for_assignment function to raise an exception:

def wait_for_assignment(consumer)
  10.times do
    break if !consumer.assignment.empty?
    sleep 1
  end
  raise TypeError.new("No assignment for consumer")
end

I also was looking for Kafka+docker issues but I had no lucky.

[thijsc]

Spent some more time on this, not getting anywhere. Tempted to get rid of the compose setup and run Kafka/Zookeeper locally again.

[thijsc]

Symptoms locally are that the connection to the broker is lost:

E, [2019-12-09T21:34:04.279632 #27004] ERROR -- : rdkafka: [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Failed to resolve 'localhost:9092': nodename nor servname provided, or not known (after 0ms in state CONNECT)
E, [2019-12-09T21:34:04.281905 #27004] ERROR -- : rdkafka: [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Failed to resolve 'localhost:9092': nodename nor servname provided, or not known (after 0ms in state CONNECT)
E, [2019-12-09T21:34:04.283745 #27004] ERROR -- : rdkafka: [thrd:localhost:9092/bootstrap]: 1/1 brokers are down

On Travis a bunch of other things break, but trying to get the build green locally first.

[thijsc]

Tried some bisection, but this is tricky since older librkafka versions don't work anymore because of the timespec linking issue. @edenhill do you have any suggestions?

[edenhill]

What's the problem? The resolution errors for localhost?
That does look odd indeed, what we've seen in other places is that the resolver bugs out if the process runs out of fds (check the fdlimit).

[thijsc]

Tried running this with wider FD limits and same result. The test suite runs fine for a while, and runs fine if you run individual tests. If you run the whole set it is unable to keep a connection after a while.

[gaffneyc]

After a bit of poking I discovered that removing the AutoPointer from Config#native_kafka gets the tests running consistently on 1.2.2 for me locally.

[thijsc]

@gaffneyc Nice find! That suggests there's something wrong with the termination of the client. Going to poke around in this direction too.

[thijsc]

@gaffneyc I'm trying using the handle without the destroy, test suite is still losing connection about half way through. Did you make any other changes?

[thijsc]

Still completely stuck on this issue, I have no leads at the moment. Behaviour I'm seeing:

• Running specs individually or in small groups succeeds.
• Running the entire consumer spec lets rdkafka lose connection to the local cluster about half-way through.

If anybody has any suggestions on in which direction to prod further I'd be very happy to hear them. @edenhill any new thoughts on this maybe?

[gaffneyc]

@thijsc No, sorry, I spent a couple hours on it one night and wasn't able to get it working. My theory was that using AutoPointer was causing issues because when objects were collected was non-deterministic and most (all?) need to be freed before the handle can be destroyed otherwise it blocks.

To test the theory I ended up starting a new client from scratch and had enough luck with it that I just... kept... going. I recently open sourced it here: https://github.com/deadmanssnitch/kafka/ and we're using it in production as of this week.

[thijsc]

Hm not super happy about that honestly, think we don't need further fragmentation and would rather welcome your contributions here.

[gaffneyc]

I'm sorry 😞 it wasn't my intention to upset you or fragment the community. It started as a proof of concept and I felt that it diverged enough from rdkafka that it made sense to be its own thing.

[thijsc]

You are of course within your rights to do whatever you want. It just seems to me that we're better off with one wrapper gem if there is no real divergence on the approach. Shall we discuss moving these two projects back into one via another channel?

In the meantime I tried removing all destroy calls to see if that fixes the consumer spec, it doesn't. So I think incorrect cleanup causing this is a red herring.

[gaffneyc]

Shall we discuss moving these two projects back into one via another channel?

Feel free to email me at my github username at gmail.

Two thoughts to quickly decide if it is AutoPointer causing issues. When a test fails run the GC then check ObjectSpace to see how many AutoPointers or other Rdkafka objects are still in memory. Second, I believe when calling rd_kafka_destroy with debug logging it will print counts of the remaining objects attached to the handle. If anything it could tell if it's worth investigating cleanup being the issue.

[thijsc]

I actually just removed all autopointers and destroy calls, thus introducing a bunch of memory leaks. Same symptoms.

[gaffneyc]

@thijsc I took a pass at removing AutoPointer from TopicPartitionList (see #107) as there are some cases where librdkafka can return a pointer to a TPL (see #subscription for example) and the application is now required to call destroy on that TPL. I'm assuming this could lead to the handle not being able to be destroyed.

I was able to get the tests all passing locally with the change.

[gaffneyc]

All passing except for one segfault where the Consumer or Producer's @native_kafka was a pointer to NULL and it tried to call rd_kafka_destroy on NULL.

[Adithya-copart]

@gaffneyc I had a similar segfault in a forked process when the autopointer releaser call to rd_kafka_destroy is made after GC'ing @native_kafka object.

I think I made some progress in fixing the build failure by switching the docker images in #106. All the jobs failed with the same two spec failures.

Update:
I'm facing the same issue in #106 now. It's before a fork call and eliminates the chances of it being GC'd in the new process which could lead to a rd_kafka_destroy call.

I wonder if rd_kafka_consumer_close free's the object first and then AutoPointer finalizer is called leading a seg fault.https://github.com/appsignal/rdkafka-ruby/blob/d0d71f669daadbfeec056085b8abe065478be784/lib/rdkafka/consumer.rb#L24

All passing except for one segfault where the Consumer or Producer's @native_kafka was a pointer to NULL and it tried to call rd_kafka_destroy on NULL.

[thijsc]

Build was fixed in #108