Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add verbose logging option to ProviderDataIngester #1420

Open
1 task
AetherUnbound opened this issue Oct 1, 2022 · 3 comments · May be fixed by #4068
Open
1 task

Add verbose logging option to ProviderDataIngester #1420

AetherUnbound opened this issue Oct 1, 2022 · 3 comments · May be fixed by #4068
Assignees
@nicoepp
Labels
🤖 aspect: dx Concerns developers' experience with the codebase ✨ goal: improvement Improvement to an existing user-facing feature good first issue New-contributor friendly help wanted Open to participation from the community 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs 🔧 tech: airflow Involves Apache Airflow 🐍 tech: python Involves Python
Projects
Openverse

Comments

@AetherUnbound
Copy link
Contributor

Description

When troubleshooting a provider API script using the new ProviderDataIngester class, it can be difficult sometimes to tell at which step in the execution an issue is happening or what the data is at that step.

It would be useful to have logging at a high level of verbosity which could be used locally in test scenarios to see what the values are at each step in the process (e.g. get_next_query_params, get_batch, get_batch_data, process_batch, etc.

Alternatives

Additional context

Implementation

  • 🙋 I would be interested in implementing this feature.
@AetherUnbound AetherUnbound added good first issue New-contributor friendly help wanted Open to participation from the community ✨ goal: improvement Improvement to an existing user-facing feature 🐍 tech: python Involves Python 🔧 tech: airflow Involves Apache Airflow 🟩 priority: low Low priority and doesn't need to be rushed 🤖 aspect: dx Concerns developers' experience with the codebase labels Oct 1, 2022
@Dkacurik
Copy link

HI,I'm willing to work on it.

@AetherUnbound
Copy link
Contributor Author

Hi @Dkacurik 😄 I'll assign the issue to you! Let us know if you have any questions about this work or getting set up 🙂

@obulat obulat added the 🧱 stack: catalog Related to the catalog and Airflow DAGs label Feb 23, 2023
@obulat obulat transferred this issue from WordPress/openverse-catalog Apr 17, 2023
@nicoepp
Copy link
Contributor

nicoepp commented Apr 8, 2024

Hi, I'm interested in implementing this feature.

@nicoepp nicoepp linked a pull request Apr 8, 2024 that will close this issue
Open
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nicoepp nicoepp

Labels
🤖 aspect: dx Concerns developers' experience with the codebase ✨ goal: improvement Improvement to an existing user-facing feature good first issue New-contributor friendly help wanted Open to participation from the community 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs 🔧 tech: airflow Involves Apache Airflow 🐍 tech: python Involves Python
Projects
Openverse Backlog
Status: 🏗 In Progress
Openverse
  
Backlog
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add verbose logging option to ProviderDataIngester nicoepp/openverse
4 participants
@AetherUnbound @obulat @nicoepp @Dkacurik