blank_as throws "invalid byte sequence in UTF-8" for string with UTF-16 LE encoding

[PikachuEXE]

File.read("/Users/pikachuexe/Downloads/test_data_feed.txt").blank_as?
# => ArgumentError: invalid byte sequence in UTF-8
require "rchardet"
::CharDet.detect(File.read("/Users/pikachuexe/Downloads/test_data_feed.txt"))
# Result
{
    "encoding" => "UTF-16LE",
  "confidence" => 1.0
}

test_data_feed.txt

[SamSaffron]

Is standard Ruby Regex able to handle this? Does stuff like length work?

[PikachuEXE]

Methods like #empty?, #length, #size works as expected
But regex matching would cause same kind of error

[PikachuEXE]

However if remove the BOM then both blank_as? and regex work

[SamSaffron]

If regex is failing can you raise this with mri at https://bugs.ruby-lang.org and link to the bug report here

…

On Wed, 4 Dec 2019 at 6:46 pm, PikachuEXE ***@***.***> wrote: However if remove the BOM then both blank_as? and regex work [image: image] <https://user-images.githubusercontent.com/1018543/70123228-28547b80-16ad-11ea-9e4d-48d8dcdb5a9c.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#27?email_source=notifications&email_token=AAABIXPWZ7IKEQNBEYD3XDDQW5N4JA5CNFSM4JUUIVPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF4BTUI#issuecomment-561519057>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAABIXPAMXJYR6PWAMZP653QW5N4JANCNFSM4JUUIVPA> .

[PikachuEXE]

Here the bug report
https://bugs.ruby-lang.org/issues/16402