Retrieve latest documents in Saved Searches

Elastic Stack Kibana
1

Hi,

I am storing JMS queue stats into ElasticSearch every minute. Each document contains "QueueName", "PendingMessageCount","EnqueuedMessageCount" and couple more stats fields. At the end of the day I wanted to be able to see in Kibana dashboard those stats, but I only wanted to see the most recent stats group by "Queue Name", and these info should not be affected by any Time Picker selection.

I went through all the visualization but found that none is suitable. I then decided to do with Saved Search. I created a search and add it to the Dashboard. I then went to edit the search object, have the following in kibanaSavedObjectMeta.searchSourceJSON:

{
  "index": "jms-stats*",
  "query": {
    "query_string": {
      "analyze_wildcard": true,
      "query": "*"
    }
  },
  "aggs": {
    "group": {
      "terms": {
        "field": "QueueName"
      },
      "aggs": {
        "group_docs": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

Unfortunately it does not seem to be taking any effect. I am still seeing records being extracted based on the time picker selection.

Could you please advise if it is possible to achieve what i want with Kibana today?

Thanks.

2

If you go back to the Discover page and load your saved search, there will be a little up arrow at the top-left of the table:

Screen Shot 2016-08-03 at 6.25.10 PM.png2152×814 115 KB

If you click this arrow, a spy panel will open. Click on the Request tab of the spy panel. This will show the actual request that is being sent to Elasticsearch. Can you copy/paste that here please? Thanks!

3

This is the request Kibana sent to Elasticsearch from the saved search:

{
  "query": {
    "filtered": {
      "query": {
        "query_string": {
          "analyze_wildcard": true,
          "query": "*"
        }
      },
      "filter": {
        "bool": {
          "must": [
            {
              "range": {
                "@timestamp": {
                  "gte": 1470207600000,
                  "lte": 1470293999999,
                  "format": "epoch_millis"
                }
              }
            }
          ],
          "must_not": []
        }
      }
    }
  },
  "size": 500,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "require_field_match": false,
    "fragment_size": 2147483647
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "30m",
        "time_zone": "America/Los_Angeles",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": 1470207600000,
          "max": 1470293999999
        }
      }
    }
  },
  "fields": [
    "*",
    "_source"
  ],
  "script_fields": {},
  "fielddata_fields": [
    "@timestamp",
    "timestamp"
  ]
}
4

Okay, so I don't think it is possible to override the aggs portion of the kibanaSavedObjectMeta.searchSourceJSON and have Discover use it. As you can see in the request Kibana actually sends to Elasticsearch, Discover adds its own aggs.

I think what you want may be covered in this issue - https://github.com/elastic/kibana/issues/6877 - but it might help me understand your use case better if you could provide a few example documents and the output you want to see from Kibana.

5

Shaunak - thanks for the info and quick turnaround on my question.

Please see below sample data that I will be indexing to ElasticSearch:

And at Kibana, when I load my dashboard, regardless of the time picker (and if possible, disregard the search criterion as well), I will be seeing only the following:

To add on, I still want to keep all the past data for histogram visualization.

6

Thanks for the examples; they helped me understand what you are going after here. It confirms what I thought earlier: you need the top_hits aggregation in Kibana to make this happen. Work for that is underway and you can follow its progress by subscribing to this issue: https://github.com/elastic/kibana/issues/6877 (same as the one in my previous comment).