Unexpected validation error with optional list field with unique_items check

[balukrishnans]

Checks

• I added a descriptive title to this issue
• I have searched (google, GitHub) for similar issues and couldn't find anything
• I have read and followed the docs and still think this is a bug

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.9.0
            pydantic compiled: True
                 install path: /home/foo/lib/python3.8/site-packages/pydantic
               python version: 3.8.5 (default, Dec 29 2021, 04:21:09)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
                     platform: Linux-3.10.0-1160.49.1.el7.x86_64-x86_64-with-glibc2.17
     optional deps. installed: ['dotenv', 'email-validator', 'typing-extensions']

Related discussion: #4050

from typing import Optional, List
from pydantic import BaseModel, Field, conlist, ValidationError

class Model(BaseModel):
    foo: Optional[List[str]] = Field(None, unique_items=True)
    bar: conlist(str, unique_items=True) = Field(None)

try:
    Model(foo=None, bar=None)
except ValidationError as exc_info:
    assert exc_info.errors() == [
        {
            'loc': ('foo',), 
            'msg': "'NoneType' object is not iterable", 
            'type': 'type_error'
        }, 
        {
            'loc': ('bar',), 
            'msg': "'NoneType' object is not iterable", 
            'type': 'type_error'
        }
    ]

Test case

def test_list_unique_items_with_optional():
    class Model(BaseModel):
        foo: Optional[List[str]] = Field(None, unique_items=True)
        bar: conlist(str, unique_items=True) = Field(None)

    assert Model().dict() == {'foo':None, 'bar': None}
    assert Model(foo=None, bar=None).dict() == {'foo':None, 'bar': None}
    assert Model(foo=["k1"], bar=["k1"]).dict() == {'foo': ["k1"], 'bar': ["k1"]}
    with pytest.raises(ValidationError) as exc_info:
        Model(foo=["k1", "k1"], bar=["k1", "k1"])
    assert exc_info.value.errors() == [
        {
            'loc': ('foo',),
            'msg': 'the list has duplicated items',
            'type': 'value_error.list.unique_items'
        },
        {
            'loc': ('bar',),
            'msg': 'the list has duplicated items',
            'type': 'value_error.list.unique_items'
        }
    ]

Fix

Update the unique_items_validator function code to.

    @classmethod
    def unique_items_validator(cls, v: 'Optional[List[T]]') -> 'Optional[List[T]]':
        if v is None:
            return v
        for i, value in enumerate(v, start=1):
            if value in v[i:]:
                raise errors.ListUniqueItemsError()

        return v

Tested this and ran all test cases and successfully passed.

Please verify.
Cc: @samuelcolvin

[balukrishnans]

Shall I raise one PR to 1.9.X-fixes

[samuelcolvin]

What happens on 1.8.2?

[balukrishnans]

There is no support for unique_items in pydantic version 1.8.2

class Model(BaseModel):
    foo: Optional[List[str]] = Field(None, unique_items=True)

Model(foo=['a','a','a'], bar=None)

Won't raise any issues (duplicate items are there in the list) and

class Model(BaseModel):
    bar: conlist(str, unique_items=True) = Field(None)

Model(bar=['a','a','a'])

Will raise an issue

Traceback (most recent call last):
  File "/Users/developer/test.py", line 1, in <module>
    class Model(BaseModel):
  File "/Users/developer/test.py", line 2, in Model
    bar: conlist(str, unique_items=True) = Field(None)
  File "pydantic/types.py", line 509, in pydantic.types.conlist
TypeError: conlist() got an unexpected keyword argument 'unique_items'

ps: I need the support for unique_items.

[samuelcolvin]

Sorry, my bad. 🙈

Yes, pr against that branch please.

[mfulgo]

Did this PR ever get made? If not, I'd be happy to create it.

[Kludex]

• It looks like this was fixed on Allows Optional lists with unique_items check #4568.

The keyword fix doesn't automatically close the issues.