Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to import images from tar using stream #380

Open
x1a0b0 opened this issue Mar 4, 2024 · 2 comments
Open

How to import images from tar using stream #380

x1a0b0 opened this issue Mar 4, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@x1a0b0
Copy link

x1a0b0 commented Mar 4, 2024

/// let bytes = byte_stream.next().await.unwrap().unwrap();

The issue: byte_stream.next()only read the first frame(the default capacity is 8k), not the complete content for large tar file.

Think aloud, import_image use Bytes as root_fs's type, which means we should read the tar file into memory one time, If the file is several gigabytes, this will be annoying.

@fussybeaver
Copy link
Owner

Yes, I believe it's bounded by 8k due to tokio's default Framing configuration ... happy to take any PR adjustments to make this better.

Re. your second point, it got changed to Bytes because Hyper changed Body to become less dependent on Tokio. Is there an obvious streaming byte implementation to use here, one that also doesn't use Tokio ?

@x1a0b0
Copy link
Author

x1a0b0 commented Mar 4, 2024

Maybe just use http-body-util's StreamBody

@fussybeaver fussybeaver added the enhancement New feature or request label Mar 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants