New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
python-bindings: Add pickle
support for G1Element
#265
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally the failure cases should be done exercised in the tests as well. Like, too few bytes, a 0-tuple and a 2-tuple, an invalid G1 point.
seed = bytes([ | ||
0, 50, 6, 244, 24, 199, 1, 25, 52, 88, 192, 19, 18, 12, 89, 6, | ||
220, 18, 102, 58, 209, 82, 12, 62, 89, 110, 182, 9, 44, 20, 254, 22 | ||
]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With this change, This binary format is part of our state that would need to be migrated if it ever changes. Is pickle a stable format? If it can change between python versions it seems risky to depend on.
There should probably be a comment here saying that these bytes may not change. If they do we'll have a migration task as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is only used as parameter for AugSchemeMPL.key_gen
to generate a PrivateKey
. I don't understand your concerns about changing bytes here. Can you elaborate?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, I didn't read this very carefully :/
I was reading this as the pickled bytes and the test to ensure it was de-pickled correctly. Since the binary format for this needs to be stable, I think a test like that would be good. It would alert us of any changes that alters the file format.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm dropped it again because it appears to create different results on CI for whatever reason.. i spent too much time with this stuff already and i don't think its super important to have this test but if you have something which works locally and on CI feel free to provide a commit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You might consider a few more tests of round-trips, especially of the infinity
value (since it serializes in a unique way).
20d3768
to
4740720
Compare
This is needed for plots caching in `chia-blockchain`.
4740720
to
50b4ff6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not an expert on pybind
, so I can only say this looks reasonable.
A couple comments about using picking in general: it gives me a bad feeling to have chia-blockchain
depend on yet another serialization mechanism, especially one so python-specific, and especially that it's used in persistence. At least it's just cache, so if we later reimplement this part in another language that doesn't easily support pickled data structures, we could get away with just ditching the python-based cache and rebuild it (but we'd likely use the Streamable
binary or JSON form at that point anyway, so why not just do that now?).
Also, I'm a bit puzzled by the fact that G1Element
doesn't already support pickling. Aren't we pickling a large number of our data structures when we pass them across the multiprocessing pool boundary? How is that working? (In this case, I'm slightly less worried about pickling, since it's a private implementation detail.)
My concerns above notwithstanding, I still approve this PR since it seems unlikely that adding pickle-ability has any downsides even if we don't end up using it ourselves. |
…etwork#265)" This reverts commit 616230c.
This is needed for plots caching in `chia-blockchain`.
…etwork#265)" (Chia-Network#278) This reverts commit 616230c.
This is needed for plots caching in
chia-blockchain
.