[WIP] Reject callback with :string return type

[larskanis]

Callbacks returning a :string type were not supported so far. It was possible to define such a callback, but the value returned was NULL in any case. This implementation rejects :string return type of callbacks.

The reason for the reject is the following: In contrast to :string parameters to called functions there is no well defined ownership of the memory of the string returned by callbacks. Instead an explicit FFI::MemoryPointer or similar should be used, which allows to track the validity of underlying memory instead of relying on some Ruby implementation details.

Fixes #751
Alternative to #770

[eregon]

The check in this PR doesn't seem correct because it prevents:

attach_function :ptr_ret_pointer, [ :pointer, :int], :string

and so tests fail.

What's the difference between a function returning a string and a callback returning a string? (it's late, I might miss something obvious)

[eregon]

What's the difference between a function returning a string and a callback returning a string? (it's late, I might miss something obvious)

Got it. If it's just a normal call, we can just read the C String to a Ruby String with

FFI::Pointer.new(address).read_string_to_null

and not have to worry what's the lifetime of that pointer.
(Well, since FFI doesn't return the pointer, there better be a separate way to free it)

But for a callback, we have a Proc returning a Ruby String, and we need to give it as char* which is indeed is quite brittle, if e.g. the Ruby String is not kept alive, or the MRI GC decides to move the String pointer. Maybe RSTRING_PTR pins the object so it can actually not move after RSTRING_PTR?

[eregon]

This seems fine to me and is safe.
And we can always evolve into allowing more (e.g., something based on strdup), but not the other way around for compatibility.

[larskanis]

@eregon Thank you for looking into the PR! I'm still busy in teaching my children as a second job due to closed schools due to the corona lockdown since 10 weeks. That's why my responses are so sparse.

[eregon]

FYI: I applied the same change in TruffleRuby master.