File encoding not properly detected causing internal server error in previewer.

[lnielsen]

Package version (if known): v2.1.0

Describe the bug

Encoding of file content not properly detected causing the preview to break.

The error only occurs in some positional placements as the detect_encoding() only reads first 1024 bytes.

Proposed solution is to default to UTF8 even if detect_encoding comes up with ascii.

Steps to Reproduce

Example file causing the error is attached when uploading it and trying to preview it.

Expected behavior

As minimum a message should say the file is not previewable, instead of an internal server error.

Screenshots (if applicable)

Additional context

Stacktrace

Markdown previewer:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1769: ordinal not in range(128)
  File "flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "invenio_app_rdm/records_ui/views/decorators.py", line 111, in view
    return f(**kwargs)
  File "invenio_app_rdm/records_ui/views/decorators.py", line 127, in view
    return f(**kwargs)
  File "invenio_app_rdm/records_ui/views/decorators.py", line 200, in view
    return f(**kwargs)
  File "invenio_app_rdm/records_ui/views/decorators.py", line 261, in view
    return f(**kwargs)
  File "invenio_app_rdm/records_ui/views/records.py", line 287, in record_file_preview
    return plugin.preview(fileobj)
  File "invenio_previewer/extensions/mistune.py", line 39, in preview
    content=render(file),
  File "invenio_previewer/extensions/mistune.py", line 25, in render
    result = mistune.markdown(fp.read().decode(encoding))

JSON previewer:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 18515: ordinal not in range(128)
  File "flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "invenio_app_rdm/records_ui/views/decorators.py", line 111, in view
    return f(**kwargs)
  File "invenio_app_rdm/records_ui/views/decorators.py", line 127, in view
    return f(**kwargs)
  File "invenio_app_rdm/records_ui/views/decorators.py", line 200, in view
    return f(**kwargs)
  File "invenio_app_rdm/records_ui/views/decorators.py", line 261, in view
    return f(**kwargs)
  File "invenio_app_rdm/records_ui/views/records.py", line 287, in record_file_preview
    return plugin.preview(fileobj)
  File "invenio_previewer/extensions/json_prismjs.py", line 57, in preview
    content=render(file),
  File "invenio_previewer/extensions/json_prismjs.py", line 26, in render
    file_content = fp.read().decode(encoding)

File

The file includes a µ on line 365.

ifdo-v2.0.0.json