Fix cli default external encoding

[casperisfine]

I believe this might be a MRI bug, but it's not confirmed yet. Either way even if it's confirmed we'll need a workaround for a while.

When using bootsnap precompile in some docker containers, the source files are improperly parsed, e.g:

APPROXIMATIONS = {
  "Þ" => "Th",
  "ß" => "ss",
}
p APPROXIMATIONS

root@f7234bf937db:/app# ruby /tmp/utils.rb 
{"\u00DE"=>"Th", "\u00DF"=>"ss"}
root@f7234bf937db:/app# ruby -e 'RubyVM::InstructionSequence.compile_file("/tmp/utils.rb")'
/tmp/utils.rb:2: warning: key "" is duplicated and overwritten on line 3
Traceback (most recent call last):
	1: from -e:1:in `<main>'
-e:1:in `compile_file': /tmp/utils.rb:2: invalid multibyte char (US-ASCII) (SyntaxError)
/tmp/utils.rb:2: invalid multibyte char (US-ASCII)
/tmp/utils.rb:3: invalid multibyte char (US-ASCII)
/tmp/utils.rb:3: invalid multibyte char (US-ASCII)

It appears that InstructionSequence.compile_file doesn't properly default to UTF-8 for source file encoding like the main parser does. On some containers I've managed to fix it with LANG=en_US.UTF-8, but on some others it don't work.

I suppose there's an inconsistency on how Encoding.default_external is initialized (cc @XrXr @tenderlove).

In the meantime this can be easily worked around by setting Encoding.default_external if it is nil

[tenderlove]

It appears that InstructionSequence.compile_file doesn't properly default to UTF-8 for source file encoding like the main parser does.

This doesn't surprise me and also it sounds like a bug in MRI

[casperisfine]

Hum, weird. When I was reproducing locally , setting default_external worked, but testing on our CI against this branch I still have the issue. I'll have to dig this more.

[casperisfine]

So I've identified the MRI bug and wrote a failing test case: Shopify/ruby@8b001a0

The issue is that Kernel.load use the command line arguments to decide the encoding, whereas InstructionSequence.compile_file doesn't. However it's not trivial to fix.

[casperisfine]

Issue opened upstream: https://bugs.ruby-lang.org/issues/17308