Set time range to "fit to data"?

(Graham Hannington) #1

I'm revisiting the Elastic Stack after several months on other work. I'm catching up; I might have missed some new stuff in recent releases.

In some index patterns I work with, the data is continuous, starting from the time an automated log forwarding process began.

However, in other index patterns, the data is in a fixed, "ad hoc" time range. For example, the data might be from a customer who had a problem, and they sent a snapshot of a log file, with a time range that includes the time when that problem occurred. That could mean an index pattern that spans a few microseconds, seconds, minutes, or even hours before and after the "actual" problem time (symptomatic log record), depending on a variety of factors. Or the data could be generated in-house by a test case.

If you're prone to thinking "TL;DR", skip this paragraph. Handling such ad hoc time ranges is complicated by the fact that, for some log records, the time stamp field might fall outside of the range of the requested snapshot. Yes, I know that sounds strange. Here's how that happens, deliberately without going into detail about the specific platform or log type: the standard utility for extracting log records for this log type accepts a start and stop time. Some of the records in the log contain records for multiple "transactions", where each transaction has its own start and stop time. For the purpose of analyzing individual transactions, a second utility program splits the original "parent" log record into one record per transaction. For reasons that I won't go into here, the time stamp for those new per-transaction records is the transaction start time. For some long-running transactions, that start time might be earlier than the start time of the snapshot specified to the first utility.

Anyway, I found myself wanting an easy way to specify to Kibana a time range that covers all of the data in a particular index pattern.

I couldn't see how to do that using the Kibana (4.x) time filter, so I wrote a tiny web UI that uses the Elasticsearch API. I call it "Stretch".

You enter the details of the index pattern:

Stretch displays the earliest and latest time stamps, with a query string parameter that you can paste directly into a Kibana URL:

I've just tried using Stretch with Elastic 5.0, and got an error 400 with the following reason:

The parameter [fields] is no longer supported, please use [stored_fields] to retrieve stored fields or _source filtering if the field is not stored

(The request URL path was /fuw-cics-*/_search?fields=@timestamp&sort=@timestamp:asc&size=1)

Okay, so, I have some catching up to do with the API.

Before I do that, though: I developed Stretch as a quick throwaway, thinking that this was likely to be a common use case, and the Kibana time filter would support this, probably sooner rather than later. Am I missing something? Can I specify "fit to data" (or equivalent) in the Kibana time filter? Or does Stretch still have legs? :wink:

(Thomas Neirynck) #2

Hi @GrahamHannington, thanks for the clear explanation.

Stretch looks like a very useful tool! Kibana does not do this. As for the API change, Kibana does not have a query-string API, so it is risky to rely on this. That said, it doesn't really seem that error after upgrading is due to changes in the query-string API.

I would instead create an enhancement request in the Kibana-repo: If this is functionality you would like to contribute to Kibana, you could consider submitting a pull request. But start with the enhancement request. That way too, we'll get some better feedback from people who are more intimately familiar with the timepicker component.

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.