New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stream add-source file upload #39
Conversation
c27384d
to
b828fd8
Compare
b828fd8
to
7a1dbd4
Compare
tilesets/scripts/cli.py
Outdated
@@ -303,35 +302,41 @@ def validate_source(features): | |||
@cli.command("add-source") | |||
@click.argument("username", required=True, type=str) | |||
@click.argument("id", required=True, type=str) | |||
@cligj.features_in_arg | |||
@click.argument("filename", required=True, type=str) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Gerdie using @cligj
means that we both:
- Can handle ld and non ld geojson (iow abstract this for the user)
- Validate and parse the geojson before passing in
I think we'll have to find a way to stream from input vs from a file. I think we have a few options:
- Stream in to temp file, stream temp file to upload
cligj
evaluates lazily so directly streaming in chunks (probably would be the weirdest to implement but most efficient overall)- ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd really like to keep cligj
if possible, it handles a lot of abstraction for users and is a good "guard" for any programs that implement reading GeoJSON.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great, except that we'll need to add the requests_toolbelt dependency in setup.py
ee30e21
to
487abe5
Compare
streaming to avoid issues related to running out of memory: #31