I'm revisiting the Elastic Stack after several months on other work. I'm catching up; I might have missed some new stuff in recent releases.
In some index patterns I work with, the data is continuous, starting from the time an automated log forwarding process began.
However, in other index patterns, the data is in a fixed, "ad hoc" time range. For example, the data might be from a customer who had a problem, and they sent a snapshot of a log file, with a time range that includes the time when that problem occurred. That could mean an index pattern that spans a few microseconds, seconds, minutes, or even hours before and after the "actual" problem time (symptomatic log record), depending on a variety of factors. Or the data could be generated in-house by a test case.
If you're prone to thinking "TL;DR", skip this paragraph. Handling such ad hoc time ranges is complicated by the fact that, for some log records, the time stamp field might fall outside of the range of the requested snapshot. Yes, I know that sounds strange. Here's how that happens, deliberately without going into detail about the specific platform or log type: the standard utility for extracting log records for this log type accepts a start and stop time. Some of the records in the log contain records for multiple "transactions", where each transaction has its own start and stop time. For the purpose of analyzing individual transactions, a second utility program splits the original "parent" log record into one record per transaction. For reasons that I won't go into here, the time stamp for those new per-transaction records is the transaction start time. For some long-running transactions, that start time might be earlier than the start time of the snapshot specified to the first utility.
Anyway, I found myself wanting an easy way to specify to Kibana a time range that covers all of the data in a particular index pattern.
I couldn't see how to do that using the Kibana (4.x) time filter, so I wrote a tiny web UI that uses the Elasticsearch API. I call it "Stretch".
You enter the details of the index pattern:
Stretch displays the earliest and latest time stamps, with a query string parameter that you can paste directly into a Kibana URL:
I've just tried using Stretch with Elastic 5.0, and got an error 400 with the following reason:
The parameter [fields] is no longer supported, please use [stored_fields] to retrieve stored fields or _source filtering if the field is not stored
(The request URL path was /fuw-cics-*/_search?fields=@timestamp&sort=@timestamp:asc&size=1)
Okay, so, I have some catching up to do with the API.
Before I do that, though: I developed Stretch as a quick throwaway, thinking that this was likely to be a common use case, and the Kibana time filter would support this, probably sooner rather than later. Am I missing something? Can I specify "fit to data" (or equivalent) in the Kibana time filter? Or does Stretch still have legs? 