ArgumentError: invalid byte sequence in UTF-8

[stephenlawrence]

When I try to parse a document with non UTF-8 characters I get this:

irb(main):283:0> content = Plaintext::Resolver.new(file, document_file.upload_content_type).text
ArgumentError: invalid byte sequence in UTF-8
from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/activesupport-4.1.7/lib/active_support/multibyte/chars.rb:172:in codepoints' from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/activesupport-4.1.7/lib/active_support/multibyte/chars.rb:172:in compose'
from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/plaintext-0.1.0/lib/plaintext/resolver.rb:37:in text' from (irb):283 from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/railties-4.1.7/lib/rails/commands/console.rb:90:in start'
from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/railties-4.1.7/lib/rails/commands/console.rb:9:in start' from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/railties-4.1.7/lib/rails/commands/commands_tasks.rb:69:in console'
from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/railties-4.1.7/lib/rails/commands/commands_tasks.rb:40:in run_command!' from /var/deploy/sdm/web_head/shared/bundle/ruby/2.1.0/gems/railties-4.1.7/lib/rails/commands.rb:17:in <top (required)>'
from bin/rails:8:in require' from bin/rails:8:in

[jkraemer]

what kind of document is that? Plaintext or CSV? In that case it's quite clear because the PlaintextHandler assumes UTF-8 as input encoding. We might either have to introduce a way to specify the encoding, leaving it up to the user to find out what that actually is for a given file, or we need some way to make an educated guess.

[stephenlawrence]

@jkraemer In this particular case its a MS Word document. Also happens with PDF.

[stephenlawrence]

Got this with a tiff file as well:

content = Plaintext::Resolver.new(file, df.upload_content_type).
Tesseract Open Source OCR Engine v3.03 with Leptonica
Cannot open input file: -dutf-8

[jkraemer]

would you mind providing sample files to reproduce these errors (the utf8 error as well as the tesseract error)?