.toUTF8 vs .fromUTF8 behavior is inconsistent, confusing and inefficient

[kyegupov]

WASM is a low-level virtual machine, so it should be able to handle strings represented as binary arrays.
There are handy methods .fromUTF8 and .toUTF8: https://github.com/AssemblyScript/assemblyscript/blob/master/std/assembly/string.ts#L499

However, they are non-symmetrical in three ways:

• fromUTF8 requires the knowledge of the string length, while toUTF8 does not return that value. You have to call lengthUTF8 separately, which is wasteful (it's already called in toUTF8)
• fromUTF8 handles \0 unicode characters in strings correctly, while toUTF8 gives an impression these are not supported
• fromUTF8 requires pure size of encoded string in bytes, while lengthUTF8 returns size with zero byte padding. Even existing tests have to adjust for that explicitly:

  assemblyscript/tests/compiler/std/string-utf8.ts

  Line 24 in b7e7be2

  assert(String.fromUTF8(ptr, len - 1) == str);

This is an inefficient and confusing approach. If the goal of AssemblyScript is to be a high-level WASM-friendly language, then having C-isms in the standard library like naked pointers to null-terminated strings feels like going against those goals.

My suggestion would be to rename .lengthUTF8 and .toUTF8 to .lengthUTF8ZeroTerminated, .toUTF8ZeroTerminated and introduce .toUTF8Buffer which returns an ArrayBuffer populated with the correct content and size. This API will be far more clear and convenient for users.

[kyegupov]

Alternative solution would be to introduce a base type like

class MemSlice {
    constructor(readonly offset: usize, readonly length: usize) {}
    ...
}

which will be quite useful in general

[dcodeIO]

Agreed, these APIs aren't ideal. Might even make sense to move them out of the string class to something specifically targeting interop (with C).

[MaxGraey]

UTF8/UTF16 api was improved in this PR