-
-
Notifications
You must be signed in to change notification settings - Fork 147
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot handle None
in custom data dumper
#377
Comments
Hello, By design, dumpers only dump bytes, and None is the only thing that could be supposedly passed as NULL. Said that... I think that there is no problem actually dumping NULL from a non-None object. I don't know about the COPY path but in the query path it should mostly work. The types annotations are definitely wrong throughout the codebase, but we can make some experiments and enlarge what dumpers can return. The way I envision it is that a dumper would just return None. If you |
I am not sure what those paths are actually 😓 As for your suggestion, if I got it right, it seems it doesn't like the class ExceptionDateDumper(DateDumper): # From Python to DB
def dump(self, obj):
year = obj.year
month = obj.month
day = obj.day
if year is None and month is None and day is None:
return None
if year < 0:
bc = " BC"
year*=-1
else:
bc = ""
day = 1 if day is None else day
month = 1 if month is None else month
return bytes(f'{year:04}-{month:02}-{day:02}{bc}', encoding='utf-8')
I had to skip some parts from the trace, so if it's not useful like that, I will try to isolate the case in a completely standalone project where it will be easier to clone and test from you too. Also tried to override the And tried inheriting directly |
Maybe this is good enough for demonstration and testing? |
Yes, this is the way I think a dumper should be written, but yes, I understand it doesn't work :) Managing None as a return value for dump() is something to be tested. Indications where things may break can be obtained by changing the dump() return value: psycopg/psycopg/psycopg/abc.py Line 103 in 8a44473
to be |
Even something simpler, but to be added to the test suite. For instance testing with a string dumper subclass dumping the empty string to NULL, or a date dumper subclass dumping all 2020 dates to NULL. |
Hi @dvarrazzo, can I try working on this feature? |
I started working on this feature in the none-returning-dumper branch. This was done a few months ago so the branch is likely stale. I've now opened MR #642. If you want to complete this work, please rebase this branch on master and add the missing parts (for instance, this branch touches all the dumpers, but some dumpers were added afterwards, e.g. numpy dumpers). Other problems I see immediately in the CI run triggered by the MR (black failing, C extension not compiling) are probably solved by changes currently in master so they might be fixed by rebasing. |
Python: 3.10.6
Psycopg: 3.0.16
I was handling dates with a custom Loader and Dumper with psycopg3:
I know the code is kinda sloppy, but the problem is that I cannot give
None
tosuper().dump()
Also if I do
return b'\N'
instead which should be the Postgres Null string, it gets converted to\\N
when given to the DB (the rest of the null values that I do not try to mess with are still just\N
so are handled fine by Postgres). So I assume it tries to escape my value that I pass while I though writing a Dumper like this would not try to do any further processing.I also tried with
\0
for good measure, but of course that didn't work.In the end I ended up doing this:
I don't like this approach though as I handle None outside the DateDumper.
If I have missed something, hopefully with your help we can update the documentation to make this more clear how it should be handled.
Thank you so much for making this project! :)
EDIT:
I also have a
__nonzero__
method in the custom type, but it does not change the handling.The text was updated successfully, but these errors were encountered: