Support regex pattern in SFTPHOOK #15409
Conversation
|
@mik-laj The author of this issue prefers to support |
|
I think, we should use fnmatchr because other operators also uses this function. This allows you to standardize the experience of using different operators so that you don't have to think about the parameter format. |
|
It would be great if your methods were similar to the SFTP for GCS operator. |
| def retrieve_file(self, remote_full_path: str, local_full_path: str) -> None: | ||
| def retrieve_file( | ||
| self, remote_full_path: str, local_full_path: str, regex_pattern: Optional[str] = None | ||
| ) -> None: |
There was a problem hiding this comment.
One thing I really dislike about this implementation is it subtly changes semantics depending on the arguments. In this function, if regex_pattern is None, remote_full_path should point to a file, but when pattern matching is performed, it should point to a directory instead (and the file name matched with the pattern). This will be very unintuitive in practice and likely cause headaches to both users and maintainers.
Pattern matching is definitely a useful feature here, but I really don’t think this is how it should be done.
|
@saveriogzz Do you want to implement it? |
yes, I will work on it! |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
In many cases, users are interested in some cases of files or types, not the whole thing. for example, users want to retrieve only CSV files in the
/tmpdirectory and etc. Unfortunately,Paramikodoes not support wirelated: #15332
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.