View only last logline from all hosts

[Marcel27 (marcel)]

For a specific use case where i have 1000+ hosts using filebeat to send a logfile to ES,
i want a view in Kibana with a (filterable) listview of all hosts, with only the last logline of each host.
Is this possible, and if so, can you give me some directions how to implement this ?

[mattkime (Matthew Kime)]

Hello @Marcel27 -

Hopefully this is helpful - Is it possible to return only the most recent of 'each document'?

[Marcel27 (marcel)]

thanks, but 2 questions arise:

1: Whatever field i use for collapse (fe. host.name) i get :
"reason": "no mapping found for hostname in order to collapse on"
Which fields can i collapse on ?

fe: GET /cmndmetrics/_mapping/field/host.name returns :

    {
      "cmndmetrics" : {
        "mappings" : {
          "host.name" : {
            "full_name" : "host.name",
            "mapping" : {
              "name" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }
            }
          }
        }
      }
    }

but:

    GET cmndmetrics/_search
    {
      "size": 1, 
      "query": {
        "match_all": {}
      },
      "collapse": {
        "field": "host.name"
      }
    }

returns error with:

            "reason": {
              "type": "search_context_exception",
              "reason": "unknown type for collapse field `host.name`, only keywords and numbers are accepted"
            }

2: When i get above working in "Dev Tools", i get (long) Json struct.
How can i use this query output to show a listview of all hosts (one line per host) ?
I think i have to be somewhere else than "dev tools" for this ?

[Marcel27 (marcel)]

I realize that what i am trying to achieve may be called custom "metrics".
I have an application which produces a logfile like:

{"field1": 21, "field2": 45, "utctime": "2020-06-26 13:48:36,", "event": "test1"}
{"field1": 62, "field2": 15, "utctime": "2020-06-26 13:53:36,", "event": "test2"}
{"field1": 12, "field2": 14, "utctime": "2020-06-26 13:58:36,", "event": "test3"}

In Kibana i want a listview with latest values like:

 hostname   field1   field2 
 hostname1    12         14   
 hostname2    24         34   

Do i have to write a metricbeat module for his (seems much work for simple requirement like this)