New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot Set Index Pattern on Elasticsearch as a Log Handler #16828
Comments
@imamdigmi, could you try this in your environment? If it works, we can make the index configurable.
|
Hi @jedcunningham thanks for your suggestion, I have tried it, and it works |
Hi @jedcunningham are you working on this? if not yet, may I help to fix this issue by submitting PR? |
Hey @imamdigmi, happy to hear you want to contribute! Have at it, I'm assigning this to you. This is a decent example of places/things that will need to be touched: #14625 Also feel free to ping me on slack if you get stuck or want a tighter feedback loop! I'm more than happy to help. |
I have started working on this here: https://github.com/apache/airflow/compare/main...kouk:support-es-index-patterns?expand=1 |
Apache Airflow version: 2.0.0
Kubernetes version (if you are using kubernetes) (use
kubectl version
):Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.8-aliyun.1", GitCommit:"94f1dc8", GitTreeState:"", BuildDate:"2021-01-10T02:57:47Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Environment: -
uname -a
):Linux airflow-webserver-fb89b7f8b-fgzvv 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 GNU/Linux
What happened:
My Airflow use fluent-bit to catch the stdout logs from airflow containers and then send the logs messages to Elasticsearch in a remote machine and it works well, I can see the logs through Kibana. But the Airflow cannot display the logs, because an error:
but when I debug and use this code, I can see the logs:
and when I look into the source code of elasticsearch providers there are no definition of the index-pattern on that
airflow/airflow/providers/elasticsearch/log/es_task_handler.py
Line 216 in 88199ee
so I assume the issue is insufficient permission to scan all the indices, therefore, how can I set the index-pattern so that Airflow only reads certain indices?
Thank you!
What you expected to happen: The Airflow configuration has option to add elasticsearch index pattern so that airflow only queries certain indices, not querying all indexes on the elasticsearch server
How to reproduce it: Click log button on task popup modal to see logs page
Anything else we need to know: Every time etc
The text was updated successfully, but these errors were encountered: