add exclude to Config

[samuelcolvin]

From a user perspective I would rather add an exclude in the Config metaclass instead of passing the dict in to .json(). One of my model's fields is a Callable and I would like to call .json() on it, however I need to instead pass a custom encoder to the .json() which still puts the responsibility on the caller.

Originally posted by @enkoder in #648 (comment)

[samuelcolvin]

as per #624 we could easily add exclude to a field, or even a dict to exclude sub-fields.

[JrooTJunior]

I guess to implements this like exporting roles in Schematics
That mechanism is good for writing API's

You can simply describe all of public fields in model and inside controllers make dump in required set of fields by specifying only the role name.

For example:

class User(BaseModel):
	id: int
	login: str
	email: str
	password: str
	first_name: str
	last_name: Optional[str] = None
	articles: Optional[List[Article]] = None

	register_date: datetime.datetime
	last_login: Optional[datetime.datetime] = None

	class Config:
		roles = {
			"self": {"exclude": {"password", "register_date", "last_login"}},
			"admin": {"exclude": {"password"}},
			"public": {"include": {"id", "email", "first_name", "last_name"}}
		}

And then:

user = User(
	id=42,
	login="mylogin",
	email="email@example.com",
	password="hashed secret password",
	first_name="Some",
	last_name="User",
	articles=[Article(id=1, label="foo"), Article(id=2, label="bar"), ...],
	register_date=datetime.datetime(...),
	last_login=datetime.datetime.now()
)
user.dump(role="public")

Should return something like this:

{
	"id": 42,
	"email": "email@example.com",
	"first_name": "Some",
	"last_name": "User",
}

include/exclude describing mechanism can be implemented more simply, for e.g. via function include(...)/exclude(...)

[enkoder]

Wow thanks for the quick response @samuelcolvin. I have just been exploring pydantic and I really like it. Having quick responses on PR's and active development certainly makes me even more excited to adopt it. Looking forward to working with you.

[dgasmith]

+1 for this, we code it into our classes as we have "local" variables that should not exported to other objects. Allowing this in the dict() on up would be optimal.

[Bobronium]

I like this idea and love @JrooTJunior's implementation with roles
I guess I'll be able to work on a new PR with these changes very soon, right after #648 will be merged

[samuelcolvin]

I understand that roles is neat, but I think it's a whole new concept that is unnecessarily complex for pydantic. If you did want roles it would be fairly trivial to add it to config and implement your own CustomBaseModel that used it as per #684.

I think exclude and include should use the fields property already in Config and also be configurable on Schema (soon to be Field, #577).

Combining this with #624, we'd have something like:

class User(BaseModel):
    id: int
    username: str
    password: str

class Transaction(BaseModel):
    id: str
    user: User
    value: int

    class Config:
        fields = {
            'value': {
                'alias': 'Amount',
                'exclude': ...,
            },
            'user': {
                'exclude': {'username', 'password'}
            },
            'id': {
                'dump_alias': 'external_id'
            }
        }

[xspirus]

Hello @samuelcolvin . First of all, I am really amazed by pydantic and, even though I haven't been using it for a long time, I think it can be really helpful in numerous situations.

On the point of this issue, as I understand, this has not been implemented yet, correct? Is there currently any work going on for this? If not, I am willing to have a go on this. Having include and exclude on Field and Config, in the way you described above, sounds like a really nice feature.

[mtucker502]

I solved this today with the following:

class Resource(BaseModel):
    class Config:
        include: Set[str] = set()
        exclude: Set[str] = set()

    def json(self, **kwargs):
        include = kwargs.pop('include', set())
        include = include.union(getattr(self.Config, "include", set()))
        if len(include) == 0:
            include = None

        exclude = kwargs.pop('exclude', set())
        exclude = exclude.union(getattr(self.Config, "exclude", set()))
        if len(exclude) == 0:
            exclude = None

        return super().json(include=include, exclude=exclude, **kwargs)

Just specify the include/exclude fields as a part of the Class definition:

class SR(Resource):
    synopsis: str
    session: str

    class Config:
        exclude = {'session'}

[xspirus]

@mtucker502 That is one way to solve when calling the .json method. However, I think that this issue tries to solve this kind of problem:

class User(BaseModel):
    id: int
    username: str
    password: str

class Transaction(BaseModel):
    id: str
    user: User
    value: int

    class Config:
        fields = {
            'user': {
                'exclude': {'username', 'password'}
            },
        }

And then calling the constructor would look like this.

>>> Transaction(id=1, user={"id": 1}, value=1)
Transaction(id=1, user=User(id=1), value=1)

Correct me if I'm wrong @samuelcolvin .

[Bobronium]

@xspirus, this is an interesting concept. And it would actually solve/alleviate problem with copiyng the same model with little changes for different contexts.

@enkoder, since this issue is originaly created from your #648 (comment), what do you think?

[samuelcolvin]

Hello @samuelcolvin . First of all, I am really amazed by pydantic and, even though I haven't been using it for a long time, I think it can be really helpful in numerous situations.

On the point of this issue, as I understand, this has not been implemented yet, correct? Is there currently any work going on for this? If not, I am willing to have a go on this. Having include and exclude on Field and Config, in the way you described above, sounds like a really nice feature.

No work on this AFAIK, happy to review a PR if it matches my suggestion above.

[xspirus]

No work on this AFAIK, happy to review a PR if it matches my suggestion above.

I will try to create a PR, as soon as I find some time to work on this 😄

[enkoder]

@MrMrRobat the approach that @mtucker502 took is how I would expect things to look! Defining excluded and included on the config like they did.

[daviskirk]

I'd be using this as well (or a variant of it) so I can also try and take a swing at a PR.

[phinate]

+1 for this feature! :)

[anth2o]

I started a branch to add this feature, I'll finalize it when I have the time to open a PR https://github.com/anth2o/pydantic/commits/add-config-to-exclude-fields

[PrettyWood]

@anth2o A PR is already open #2231. No need to waste your time ;) But your review or feedback on it when it's ready is always more than welcome!

[phinate]

Will the implemented behaviour in #2231 mean that recursive serialization respects the config of sub-classes for exclude?

[daviskirk]

Will the implemented behaviour in #2231 mean that recursive serialization respects the config of sub-classes for exclude?

That is definitely the goal. We will see how it works in practice

[johnsturgeon]

Has this made it's way into production yet?

[johnsturgeon]

@samuelcolvin -- PyPi does not seem to pull this change. I had to update my requirements to pull from the repo.

[PrettyWood]

It has been merged but not yet released. It will be part of 1.9

[johnsturgeon]

Thank you!

[dixonwhitmire]

Hi Folks,

Thank you very much for supporting the config model "exclude" feature in 1.9.! This is going to help my team simplify things a bit.

Is there a release cadence or timeline for the 1.9 release?
Thanks!

[ludwignyberg]

Any updates on the 1.9 release timeline?

[gouthamravella]

As inspired by @mtucker502 answer, I have solved this issue with a custom class. This class can even includes/excludes the nested model fields too with both model.dict() and model.json(). You can alternatively even pass the params directly in the dict() and json() functions as well. Please try it and let me your suggestions on this. Hope it helps to someone like me.

class Resource(BaseModel):
    class Config:
        include: Dict[str, Any] = dict()
        exclude: Dict[str, Any] = dict()
    
    def dict(self, **kwargs):
        include = kwargs.pop('include', None)
        exclude = kwargs.pop('exclude', None)

        _incl = getattr(self.Config, "include", dict())
        if _incl != None and len(_incl) > 0:
            if isinstance(_incl, set):
                include = _incl
            else:
                if include == None:
                    _inc = dict()
                    include = dict(ChainMap(_inc, _incl))
                else:
                    include = dict(ChainMap(include, _incl))
        
        _excl = getattr(self.Config, "exclude", dict())
        if _excl != None and len(_excl) > 0:
            if isinstance(_excl, set):
                exclude = _excl
            else:
                if exclude == None:
                    _exc = dict()
                    exclude = dict(ChainMap(_exc, _excl))
                else:
                    exclude = dict(ChainMap(exclude, _excl))
        return super().dict(include=include, exclude=exclude, **kwargs)
    
    def json(self, **kwargs):
        include = kwargs.pop('include', None)
        exclude = kwargs.pop('exclude', dict())

        _incl = getattr(self.Config, "include", dict())
        if _incl != None and len(_incl) > 0:
            if isinstance(_incl, set):
                include = _incl
            else:
                if include == None:
                    _inc = dict()
                    include = dict(ChainMap(_inc, _incl))
                else:
                    include = dict(ChainMap(include, _incl))
        
        _excl = getattr(self.Config, "exclude", dict())
        if _excl != None and len(_excl) > 0:
            if isinstance(_excl, set):
                exclude = _excl
            else:
                if exclude == None:
                    _exc = dict()
                    exclude = dict(ChainMap(_exc, _excl))
                else:
                    exclude = dict(ChainMap(exclude, _excl))
        return super().json(include=include, exclude=exclude, **kwargs)

Usage:

class Coords(Resource):
    lat: float
    lng: float

class Location(Resource):
    city: str
    coords: Coords

class User(Resource):
    name: str
    location: Optional[Location]
    other: Optional[str]

    class Config:
        exclude = {}
        include= {'name':...,'location': {'city':...,'coords':{'lat'}}}

data={
    "name":"name",
    "other": "other",
    "location": {
        'city': 'Hyd',
        'coords': {
            'lat': 17.4,
            'lng': 17.6
        }
    }
}

User(**data).json()
(or)
User(**data).dict()
(or)
User(**data).json(exclude={'name'})
(or)
User(**data).json(include={'other':..., 'location':{'coords':{'lat'}} })