we are encountering an issue on elasticsearch trying to display fields based on certain search criteria.
We have an index with a "payload" field which has multiple properties
What we are trying to do is to request the index to retrieve properties based on criteria.
For example, for all the logs where payload.codeClient=something, list all other payload properties ever set in messages matching that criteria.
We have tried some requests so far but none of them worked properly
GET our_alias_name/_search
{
"size": 100,
"query": {
"bool": {
"must": [
{
"range": {
"timestamp": {
"gte": "now-6M/M"
}
}
},
{
"query_string": {
"default_field": "codeMessage",
"query": "TRT*"
}
},
{
"match": {
"payload.typeObjet.keyword": "paiement"
}
}
]
}
},
"fields": [
"payload.*"
],
"sort": [
{
"timestamp": {
"order": "desc"
}
}
],
"_source": false
}
And what we got as an answer was:
{
"took" : 304,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [
{
"_index" : "index_name",
"_type" : "_doc",
"_id" : "ySzVVogB36_bMKME_Ll0",
"_score" : null,
"fields" : {
"payload.applicationCible.keyword" : [
"Kyriba"
],
"payload.codeBranche" : [
"MEF"
],
"payload.typeObjet" : [
"paiement"
],
"payload.applicationSource.keyword" : [
"SAGE MEF"
],
"payload.typeObjet.keyword" : [
"paiement"
],
"payload.applicationSource" : [
"SAGE MEF"
],
"payload.codeBranche.keyword" : [
"MEF"
],
"payload.applicationCible" : [
"Kyriba"
]
},
"sort" : [
1685084047337346939
]
},
{
"_index" : "azure_aks_container_logs",
"_type" : "_doc",
"_id" : "uhbVVogBXEyG-ruo_HKV",
"_score" : null,
"fields" : {
"payload.applicationCible.keyword" : [
"Kyriba"
],
"payload.codeBranche" : [
"RHU"
],
"payload.typeObjet" : [
"paiement"
],
"payload.applicationSource.keyword" : [
"RHPI"
],
"payload.typeObjet.keyword" : [
"paiement"
]....................
but the test above does not show a field wich has not been set in query result, and what we want to get as an answer is not an array with messages content but an array with all field names that have ever been filled and respecting the criteria payload.typeObjet=paiement
We have looked at the possibility of using aggs but as said previously, what we want as an answer is an exhaustive list of ever filled (non empty) field names matching the search criteria
