Use yapic.json instead of json for 1.5x to 2x performance gain #237
Conversation
…e bytestings manually
|
you have the wrong dependency here. It should be: https://pypi.org/project/yapic.json/ So, i dont think you need to re-test for a week, but thats definitely the wrong library. yapic installs tensoflow and a million other reqs that arent needed |
Thanks for catching the error in setup.py. We don't need tensorflow (yet) :-) |
cryptofeed/exchange/huobi.py
Outdated
| @@ -74,7 +74,7 @@ async def _trade(self, msg: dict, timestamp: float): | |||
|
|
|||
| async def message_handler(self, msg: str, timestamp: float): | |||
| # unzip message | |||
| msg = zlib.decompress(msg, 16 + zlib.MAX_WBITS) | |||
| msg = zlib.decompress(msg, 16 + zlib.MAX_WBITS).decode(encoding='utf-8', errors='strict') | |||
There was a problem hiding this comment.
@vincentmele looks like the library has been updated to do byte parsing, so you should be able to remove this and require the latest version of the library in setup.py
…te arrays instead of only strings; added minumum requirement of yapic.json 1.41 in setup.py
|
Good idea to improve the json decoding. How does yapic.json perform compared to e.g. ultrajson? |
ultrajson doesn't support "parse_float=decimal.Decimal", which is required. I opened the issue here: ultrajson/ultrajson#401, then found that yapic supported it and was extremely fast. |
|
@dikkechill yep - the big requirement here is parsing floats to decimal.Decimal, I haven't found any other libs that support this, but I also haven't exhaustively searched either |
Documented in #231 .
Tested for a week continuously (on only a couple exchanges, including Huobi) and I see no abnormal results.