1 sentence utf-8 detected as Windows-1252

[EralpB]

Thanks for coming up with this utility, it's a great need but this fails even with simplest examples I can't see how I'd trust this on production.

bs_2 = b'Carter\xe2\x80\x99s Janitorial'
chardet.detect(bs_2)
{'encoding': 'Windows-1252', 'confidence': 0.73, 'language': ''}
bs_2.decode('cp1252')
'Carter’s Janitorial'
bs_2.decode('utf-8')
'Carter’s Janitorial'

am I doing something wrong? it's just couple of characters and it's clearly utf-8.

[Ousret]

You can try https://github.com/Ousret/charset_normalizer :)

[yomajo]

I am working with CSV files currently and personally tried 2 detection libraries. Both failed to detect contents are windows...: cp1257

In some StackOverflow answers I found:

def get_encoding(csvfile_path):
with open(csvfile_path) as f:
print(f)

which outputs:
<_io.TextIOWrapper name='Output/Headlines_data.csv' mode='r' encoding='cp1257'>

So I wrote basic extractor myself, with a fallback to lib solution:
from re import findall
def get_encoding(csvfile_path):
with open(csvfile_path) as f:
raw_result = str(f)
try:
matches = findall(r"encoding='.+'>", raw_result)
encoding = matches[0].replace('encoding=\'', '').replace('\'>','')
return encoding
except:
return get_encoding_via_lib(csvfile_path)

I'm kinda new, sorry for indentation stuff, code block ignores my spaces...

[chomechome]

I also haven't been able to successfully use chardet in production. So, I've recently built a more accurate encoding detection library myself: https://github.com/chomechome/charamel