Add support for custom encoders

[thedrow]

I need custom encoders because I'd like to integrate it with Django-Rest-Framework.
See encode/django-rest-framework#1255 for details.

[jskorpan]

No it does not. It’s a major feature we envision for a possible future ujson 2.0
//JT

[thedrow]

What exactly needs to be done in order to progress this further?

[jskorpan]

A speedy method of identifying and specifying custom encoders has to be defined and implemented. I’ve done some thinking around this, either we just have one custom encoder function which all values are passed through or we could go more complex and have a Dict<pythonClass, handleFunc> data structure. This would also allow us to modularize some rather type specific encoders we currently have in the code which may or may not serialize data to everyone’s liking.

I also recall an issue with how we are able to handle custom handlers returning a type which is in term is valid JSON. I can’t remember the exact details but my conclusion was to do a rewamp of the encoder step to fully support this better, there was something fundamental.

//JT

[thedrow]

Don't you think your API should be compatible with the standard json module?

[jskorpan]

I suppose we should honor the “drop in replacement” as far as possible yes.
We could add the class -> func mapping as an internal API not visible externally or as an extension on top of the JSON module spec.

[thedrow]

I need a class encoder in order to drop in ujson in DRF. See https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/utils/encoders.py#L15

[jskorpan]

Moving this into the possible 2.0 milestone for ujson

[thedrow]

What's the status on this? How can I help make it happen?

[thedrow]

ping?

[nburns]

I would love to be able to use custom encoders 👍

[lepture]

👍

[suzaku]

👍

[Zuckonit]

[cilia]

And it'd be nice to have support for 'default' handler: default(obj) is a function that should return a serializable version of obj or raise TypeError. The default simply raises TypeError.

[zllak]

👍

[faizandfurious]

👍

[minglecm]

👍

[cristianocca]

I would love this feature as well, ujson is still faster than python's json even after 2.7 which moved into C code. What I don't know, if enabling custom encoders will slow down serialization to a point where ujson is as fast (or slow) as default python 2.7 json encoder.
What bothers me the most, is that even if is implemented in C, it is still 3 times as slow as nodejs json serialization which is also implemented in C

[dobber1134]

👍

[nryoung]

And it'd be nice to have support for 'default' handler: default(obj) is a function that should return a serializable version of obj or raise TypeError. The default simply raises TypeError.

100% agree on this. I would love to switch over to ujson completely but ujson doesn't support nested iterable data types.

[mayerwin]

👍👍👍 Please also support custom encoding of dictionary keys, not just values. This is missing in all the json libraries I have tried, for no good reasons IMHO (performance impact should be negligible for those who would not use it). ujson is currently the best I have found thanks to the __json__ attribute (though I believe a handler approach, as jsonpickle is doing for example, would allow a cleaner and more decoupled extensibility).

[gjuarezh]

add support please!!

[megahoy]

+1 for this issue
need this feature

[lvalladares]

This would be a really nice feature, actually im having to switch back to standard json because my dict has some custom classes objects

[jz1371]

+1 for adding this feature 👍

[olalonde]

Is there a workaround for this? I need this since some of my dicts contain numpy floats.

[mitar]

So you saw that you can use __json__ method on an object to customize encoding to JSON of your values? See here example.

[olalonde]

Ah thanks @mitar . I still ended up writing this workaround since I don't always have control over the objects I'm jsonifying:

def is_iterable(x):
    try:
        iter(x)
        return True
    except TypeError:
        return False


def map_anything(x, fn):
    if isinstance(x, str):
        return fn(x)
    if isinstance(x, dict):
        return {k: map_anything(v, fn) for k, v in x.items()}
    if is_iterable(x):
        return [map_anything(ele, fn) for ele in x]
    return fn(x)


def prepare_for_json(val):
    if isinstance(val, numpy.int32):
        return int(val)
    if isinstance(val, numpy.int64):
        return int(val)
    if isinstance(val, numpy.number):
        return str(val)
    return val


def json(data, **kwargs):
    # cls not supported by ujson
    # https://github.com/esnme/ultrajson/issues/124
    return json_dumps(map_anything(data, prepare_for_json), **kwargs)

[jackton1]

Modifying @olalonde suggestion can also use the DRF encoder.

def serializer_func_mapper(x, fn):
    if isinstance(x, six.text_type):
        return six.text_type(x)
    if isinstance(x, six.integer_types):
        return int(x)
    if isinstance(x, dict):
        return {k: serializer_func_mapper(v, fn) for k, v in x.items()}
    if is_iterable(x):
        return [serializer_func_mapper(item, fn) for item in x]
    return fn(x)

 def json(data, **kwargs):
        if data is None:
            return bytes()
        encoder = JSONEncoder().default
        return ujson.dumps(serializer_func_mapper(data, encoder))

[joranbeasley]

don't the solutions listed above here totally negate any benefit derived from ujson? I would suspect at this point you would see better timing just using normal json with a default argument

[dstromberg]

I'd like cls= as well. Or something like it.

[Jeepeo]

Well even 2.0.3 is released.. What about this milestone, forgotten?

[hugovk]

I've removed it from the milestone.

[Jeepeo]

but still waiting tho ;)