You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Azure object store implementation uses default list_with_offset implementation. It lists everything and does the filtering client side.
This is problematic for Delta Lake with large directories where we only really care about listing the latest files from a certain path.
Describe the solution you'd like
Microsoft pointed me to the Hadoop Azure filesystemlistStatus API where startFrom is supported.
The downside is that this is not documented. Perhaps this could be an opt-in option if there is hesitation using an undocumented format for the token that could break.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Azure object store implementation uses default
list_with_offset
implementation. It lists everything and does the filtering client side.This is problematic for Delta Lake with large directories where we only really care about listing the latest files from a certain path.
Describe the solution you'd like
Microsoft pointed me to the Hadoop Azure filesystem
listStatus
API wherestartFrom
is supported.Basically the starting path is embedded in the opaque continuation token provided when the list API needs to page results. On the initial request this token is created and provided to the list API to start from the path.
The downside is that this is not documented. Perhaps this could be an opt-in option if there is hesitation using an undocumented format for the token that could break.
The text was updated successfully, but these errors were encountered: