Constructing a Request Using Kibana Saved Search Information

Elastic Stack Kibana
1

Hello all!

Newbie here. I am trying to build an external tool that constructs requests to Elasticsearch that will return the same data displayed in the discover tab of Kibana when a saved search is opened, and then export said data to various places.

I have set up a UI that will retrieve all of the saved searches that are in projects I have access to, but I am currently stuck trying to figure out how to use the saved search objects to construct a request that will return only the columns provided by the saved search and their timestamps.

Consider the following JSON document returned from a call in my programs API:

{
"success": true,
"data": {
    "total": 1,
    "max_score": 1,
    "hits": [
        {
            "_index": ".kibana_demo-index",
            "_type": "search",
            "_id": "Demo-Search",
            "_score": 1,
            "_source": {
                "title": "Demo Search",
                "description": "",
                "hits": 0,
                "columns": [
                    "type",
                    "AppId",
                    "LauncherAppId",
                    "Uuid"
                ],
                "sort": [
                    "#demo-timestamp",
                    "desc"
                ],
                "version": 1,
                "kibanaSavedObjectMeta": {
                    "searchSourceJSON": "{\"index\":\"[demo-index-]YYYY.MM\",\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"fragment_size\":2147483647},\"filter\":[],\"query\":{\"query_string\":{\"query\":\"*\",\"lowercase_expanded_terms\":false,\"analyze_wildcard\":true}}}"
                }
            }
        }
    ]
  }
}

The above JSON document contains the information of a saved search in a project I have access to. The search populates the discover table with 4 columns of data listed above and sorts the timestamp for the events.

  • How can I use the information from the search above to query and return a JSON document only containing the same information displayed in the discover table?

I hope that makes sense, please let me know if you need any clarification. Thank you in advance for your time!

Joey

2

Hi @JoeyMarinello,

first off, I want to point out that the way Kibana stores saved searches can and
will change in the future and does not constitute a stable api for external
use. :wink:

If you still want to go ahead, I can give some pointers:

  • the columns field contains the columns displayed
  • the sort field contains the sorting criterion in the form [column, direction]
  • the kibanaSavedObjectMeta is a JSON-encoded string containing the index pattern, the filter and the query

From these information you could create a [query](https://www.elastic.co/guide/
en/elasticsearch/reference/current/search-request-body.html) that could roughly look
like this:

GET /${INDEX_PATTERN}/_search
{
  "query": {
    "bool": {
      "must": [
        ${QUERY_FROM_JSON}
        ...${FILTERS_FROM_JSON}
        {
          "range": {
            "@timestamp": {
              "gte": ${FROM_TIMESTAMP},
              "lte": ${TO_TIMESTAMP},
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "${COLUMN}": { "order": "${DIRECTION}" }
    }
  ],
  "_source": [
    ...${COLUMNS}
  ]
}

The ${ABC}s are placeholders for the values extracted from the saved search
document. It's obviously just a rough outline that needs to be adapted to your
specific use case.

Also of interest might be to know that the ability to export discover query results as CSV without fiddling with requests yourself is being worked on and will be released as part of x-pack's reporting feature when done.

1 Like
3

Hi Felix!
Thanks for an in-depth reply. Much appreciated!
I'm pretty new to programming and elasticsearch, but I am guessing used saved search objects is unstable because the way they are stored and constructed can, and will, change in the future? Which means code will have to be updated and maintained on my end.

Another elastic team member recommended I scrape the screen when a search is loaded, or I create a plugin using your template. Are these options more viable for a longer period of time?

Thanks for the info regarding the discover tab csv change. Any rough estimate on how long until it will be released? Trying to figure out if it's worth my time to continue this project. It's definitely been a great learning experience so far!

Thanks for your help Felix, you and your team rocks!

Regards,
Joey Marinello

4

Thanks for the kind words, Joey. :slight_smile: Unfortunately using saved objects, screen scraping and writing a plugin all require maintenance on your side. 5.5 will probably introduce a dedicated ReST API for access to the saved objects (see Kibana PR #11632), but that doesn't mean the API won't change from version to version.

I don't know the exact requirements for your tool, but building something tailored to your use-case separate from Kibana might be the way to go. You can use the inspector panel in discover (the small upwards arrow beneath the histogram above the table) to observe what Kibana's requests look like. The devtools app in Kibana is very useful for experimentation with those queries.

The official CSV export feature would probably mean the least amount of work for you. It is planned for stage-by-stage rollout in the next major version (6.0) of x-pack later this year. I can not give you a definitive date for that though.

5

Felix,

Thanks, again, for all this great information. I feel like I am starting to get comfortable with Elasticsearch, yay! :slight_smile:

The project I am working on stores data in monthly indices (YYYY-MM). Upon inspecting the POST requests that return the data from Elasticsearch, I've noticed that a request is sent for each monthly index within search bounds.

Is there a way to search across multiple, or all indices? We have a timestamp field that I would like to use for search bounds, but if I cant search across all indices, I feel like it would make more sense to programmatically send a request using each index within the bounds.

Thank you for your time, hope you're having a great week!

Joey Marinello

6

Glad to hear that :slight_smile:

How Kibana treats the indices is controlled in the index pattern configuration using the settings "Expand index pattern" and "Use event times to create index names":

image.png611×734 44.3 KB

Both will be considered deprecated from 5.5 on, because Elasticsearch has become smart enough to figure out the indices from wildcards to use for efficient query processing by itself. I would therefore recommend to use patterns of the form index-* to match all indices starting with index-.

When querying from your own application you can also specify multiple indices: You can use comma-separated index names (GET /index-1,index-2/_search), wildcards (GET /index-*/_search) and various other modifiers (see the api convention docs).

1 Like
7

Awesome, I've got it working using the timestamp field!

Thank you so much for your help and patience, Felix! :slight_smile: Kind regards to you.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.