You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
how to use grafana with zincsearch using lucene queries
how to query zincsearch for logical gorups ie: a AND (b OR c)
Briefing
I don't know hot to classify this, so here are the facts:
I'm trying to use grafana (GF) with zincsearch (ZS) 0.4.8 (using the elasticsearch (ES) plugin and the /es endpoint) and I'm not getting the same results as ES.
GF uses Lucene query syntax, which works as expected in ES, suppose we have this data:
game
platform
week
God of Tar
PS2
24
Nano Droids
PS2
30
Legend of Zira
XBOX
24
Remote Remorse
XBOX
40
Grant Andreas
EPIC
20
Saint Col
EPIC
24
A Lucene query in GF like this: week:24 AND platform:(PS2 OR XBOX)
brings only 2 records in ES (as expected), but in ZS brings the whole set.
So, I tried to figure out why.. after some packet capturing, I notice that the GF plugin uses the endpoint _msearch for getting the data. With this I made a test case in order to replicate the problem, and.. boy oh boy, the problem is bigger.
The response from ZS is not the whole set, is the whole sets (yes, plural). Somehow it returns all the data in the indexes.
So basically, my main question is:
how can I use GF with ZS with Lucene query, or ...
how to write the Lucene query above to return those 2 records ONLY
How to replicate
In the test case I added a second data set as:
date
vendor
week
2023-08-13
sony
33
2023-08-13
enix
33
# Run version 0.4.8 os ZS
ZINC_FIRST_ADMIN_USER=admin ZINC_FIRST_ADMIN_PASSWORD=admin ./zincsearch
# Get the list of indexes (empty)
curl admin:admin@localhost:4080/api/index
# execute lucene query for sample, should return 2 records as ES
curl -X POST -H 'Content-Type: application/json' admin:admin@localhost:4080/es/_msearch -d '{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"sample"}{"size":500,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1672552800000,"lte":1704088799999,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"week:24 AND platform:(PS2 OR XBOX)"}}]}},"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}},{"_doc":{"order":"desc"}}],"script_fields":{}}'
{"responses":[
{"took":0,"timed_out":false,"_shards":{"total":6,"successful":6,"skipped":0,"failed":0},"hits":{"total":{"value":8},"max_score":1,"hits":[
{"_index":"sale-2023.08","_type":"_doc","_id":"218HL9uvD5C","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","date":"2023-08-13","vendor":"sony","week":33}},
{"_index":"sale-2023.08","_type":"_doc","_id":"218HL9xlCog","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","date":"2023-08-13","vendor":"enix","week":33}},
{"_index":"sample","_type":"_doc","_id":"218HKlt8lUI","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"God of Tar","platform":"PS2","week":24}},
{"_index":"sample","_type":"_doc","_id":"218HKlz5VE5","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"Legend of Zira","platform":"XBOX","week":24}},
{"_index":"sample","_type":"_doc","_id":"218HKlz5VE7","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"Grant Andreas","platform":"EPIC","week":20}},
{"_index":"sample","_type":"_doc","_id":"218HKlz5VE6","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"Remote Remorse","platform":"XBOX","week":40}},
{"_index":"sample","_type":"_doc","_id":"218HKlz5VE4","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"Nano Droids","platform":"PS2","week":30}},
{"_index":"sample","_type":"_doc","_id":"218HKlz5VE8","_score":1,"@timestamp":"2023-08-13T12:00:00Z","_source":{"@timestamp":"2023-08-13T12:00:00Z","game":"Saint Col","platform":"EPIC","week":24}}
]}}]}
# same query from grafana to ES
curl -X POST -H 'Content-Type: application/json' localhost:9200/_msearch -d '{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"sample"}{"size":500,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1672552800000,"lte":1704088799999,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"week:24 AND platform:(PS2 OR XBOX)"}}]}},"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}},{"_doc":{"order":"desc"}}],"script_fields":{}}'
options:=querystr.DefaultOptions()
// TODO fields// TODO default_field// TODO default_operator// TODO boostzer, _:=zincanalysis.QueryAnalyzer(analyzers, value.Analyzer)
TLDR;
Briefing
I don't know hot to classify this, so here are the facts:
I'm trying to use grafana (GF) with zincsearch (ZS) 0.4.8 (using the elasticsearch (ES) plugin and the /es endpoint) and I'm not getting the same results as ES.
GF uses Lucene query syntax, which works as expected in ES, suppose we have this data:
A Lucene query in GF like this:
week:24 AND platform:(PS2 OR XBOX)
brings only 2 records in ES (as expected), but in ZS brings the whole set.
So, I tried to figure out why.. after some packet capturing, I notice that the GF plugin uses the endpoint _msearch for getting the data. With this I made a test case in order to replicate the problem, and.. boy oh boy, the problem is bigger.
The response from ZS is not the whole set, is the whole sets (yes, plural). Somehow it returns all the data in the indexes.
So basically, my main question is:
How to replicate
In the test case I added a second data set as:
# Run version 0.4.8 os ZS ZINC_FIRST_ADMIN_USER=admin ZINC_FIRST_ADMIN_PASSWORD=admin ./zincsearch
# Get the list of indexes (empty) curl admin:admin@localhost:4080/api/index
# Get the list of indexes again (only 2 indexes) curl admin:admin@localhost:4080/api/index
The text was updated successfully, but these errors were encountered: