Extracting Data retrieved by Timelion

Elastic Stack Kibana
1

Hi all,
I'm new to ELK and am really enjoying it so far.

I'm working on deep learning and am interested in feeding my neural net aggregations of my raw data over some time units (e.g. if I have several days of raw data I want to split them to 15m intervals and output a sum or mvgavg of values in the interval). To test the aggregations make sense I have used Timelion and am very happy with the results I got.

My question is: Is there a way to get the "raw" values Timelion extracts rather than going through the hassle and, worse, the risk of inconsistency by writing equivalent queries to ES?
So, if I have a Timelion expression of the type:
.es(index=dataset1,timefield=time,metric=sum:field1).derivative()
Which generates some visualization, I want to get the raw data table that this query creates (i.e. the series of times and values that is used to plot the visualization) and use that data as input to my neural net. I'm assuming that in order to plot the visualization there is some sort of basic data table retrieved that looks something like:

  • 31.3.2018 13:00 >>> 5012
  • 31.3.2018 14:00 >>> 5089
  • ....

Is there a way to get this basic table somehow via a Kibana query or the exact ES query that generates this data for the visualization?

It is really important to me to ensure that there are no inconsistencies between the value I see in the visualization and the values I will use for training the neural net.

Thanks in advance!

p.s - I have tried to search for solutions to similar questions but failed to find anything concrete. I have found how to do it for other visualizations here - but that doesn't work for Timelion visualizations, and here - but I don't see how to use the values that I find there to get the data I want.

2

Update:
(I rather not edit as it has been a few hours since the original post)

I have found a workaround to get the raw requests to ES by enabling slowlog:

curl -XPUT "http://localhost:9200/dataset1/_settings" -H 'Content-Type: application/json' -d'
{
"index.search.slowlog.threshold.query.info": "0s"
}'

and then reading the "source" details.
(if you do this I highly recommend to restore original settings once you're done, otherwise you'll have a flood of logs)

The problem is that calculations such as derivative, mvgavg, etc are not reflected in the raw requests. I am assuming that those are done live in the Kibana/Timelion instance based on the results. So, I would still like to find a way to get the data table used for plotting the Timelion graph.

Any advice would be greatly appreciated.

Thanks!

3

There isn't any exporting with timelion unfortunately (I'd recommend filing an enhancement request at https://github.com/elastic/kibana/issues/new). If you want the exact data timelion uses the best way would be to use its api directly. You can reverse the endpoint by checking the network tab in your browser's developer tools and looking for api/timelion/run. The short version is it takes the query and time range, and returns x,y tuples.

If you want a more kibana specific version of the slowlog you can turn on elasticsearch.logQueries in kibana.yml. It will have the same issues you mentioned above however, with not all the calculations happening in the query.

Is that more in line with what you're looking for?

4

yes, it is - thanks a lot for your help.
I will look into filing the enhancement request if it isn't too troublesome