Question: is there a plan to support streaming from GCS?

[dnnspark]

🚀 Feature

Motivation

Pitch

Alternatives

Additional context

[github-actions]

Hi! thanks for your contribution!, great first issue!

[tchaton]

Hey @dnnspark,

Yes, this is quite simple to add. Simply needs to add the downloader.

[dnnspark]

Thanks @tchaton, do you have an idea when it's going to land (even very rough estimate)?

[tchaton]

Hey @dnnspark,

If you are willing to give it a try, I can look into it this week.

[dnnspark]

Sorry for the late @tchaton

I'm willing to try! But it's not blocking at the moment, so I will stay tuned about the GCS support (it will be very helpful if you ping on this thread once it's ready).

One thing I notice is that optimize() function assumes the data is stored on local disk (at least in the example). In my case, the raw data is at GCS (because it's too large). Is there a way to transform the data that is stored in the cloud, and save the transformed data to the cloud without having to download the entire data?

[tchaton]

Sorry for the late @tchaton

I'm willing to try! But it's not blocking at the moment, so I will stay tuned about the GCS support (it will be very helpful if you ping on this thread once it's ready).

One thing I notice is that optimize() function assumes the data is stored on local disk (at least in the example). In my case, the raw data is at GCS (because it's too large). Is there a way to transform the data that is stored in the cloud, and save the transformed data to the cloud without having to download the entire data?

Yes. that's why this library was built :) But I would need to add GCS support for it ;) I will try to prioritize it.