Group By (Aggregation) to get only latest fields value

Elastic Stack Elasticsearch
1

I write a query to read the metricbeat file. This gives me the whatever I want but it repeats the value multiple time.

I want to group by this on latest timestamp so I can get only latest record.

Below is my query

string indexName = "metricbeat-7.4.2-" + DateTime.Now.Year.ToString() + "." + DateTime.Now.Month.ToString("00") + "." + DateTime.Now.Day.ToString("00");
connectionSettings = new ConnectionSettings(connectionPool).DefaultIndex(indexName);
elasticClient = new ElasticClient(connectionSettings);

        string[] systemFields = new string[]
        {
            "system.memory.actual.used.pct",
            "system.cpu.total.norm.pct"                
        };

        var elasticResponse = elasticClient.Search<object>(s => s
            .DocValueFields(dvf => dvf.Fields(systemFields))
            );

DSL query

get /metricbeat*/_search?pretty=true
{
"query" : {
"match_all": {}
},
"docvalue_fields" : [
"system.memory.actual.used.pct",
"system.cpu.total.norm.pct",
"system.load.5",
"docker.diskio.summary.bytes"
]
}

2

Please help

3

please note that this is a volunteer driven forum, and thus does not come with any guarantee of questions answered, so you may want to wait a little bit longer than a few hours before bumping your message. if you need support, it is there :slight_smile:

For each metric you could execute a search with a size of 1 document to be returned and sort your search.

Or you could do a single search with for max aggregations for each field.

hope this helps!

4

Thanks Alexander for your suggestion and solution. I know this foram is volunteer but I was very very impatient this time. Sorry for that.

Any sample for second approch will be more helpful if I am not much bothering you.

5

I tried this but it gives me 6 hour old record. like if I hit @11Am then it shows me record of 5AM

GET /metricbeat-7.4.2-2020.01.10/_search?pretty=true
{
"size": 0,
"aggs": {
"memory_aggs": {
"terms": {
"field": "system.memory.actual.used.pct",
"size": 1
},
"aggs": {
"max_timestamp": {
"max": {
"field": "@timestamp"
}
}
}
}
}
}