Kibana Timestamp Search

Elastic Stack Elasticsearch
1

I have daily indexes configured in ElasticSearch and an ALIAS which points to the last 7 days index.
In Kibana I have configured this Alias but whenever I search lets say last 24 hours its searching against all the Indexes present in that Alias.

Is it an expected behavior.

2

Possibly, but like all good answers, it depends ;).

What's the index pattern you are using in Kibana, what's the alias index, and what are the daily indices?

I ask because if the names of the daily indices and weekly index overlap with the provided index pattern in Kibana, then yes, it will search against both the originals and the alias. For example, if you have logs-2016.10.[23-29] and you wrap them up into logs-2016.10, and then in Kibana you use logs-*.

3

Its just daily indexes for e.g. logs-2016.10.[1-30]. The alias name is just logs which is configured in Kibana as well. Ideally I thought it would search against the alias but if I give 24 hours, it would just search the current/last index and not all the indexes present in the alias even 5 days old for e.g.

4

So you have an alias index named logs and you've simply used logs as the pattern in Kibana, is that correct?

I think what happens behind the scenes is that Elasticsearch will query all of the daily indices that belong to that alias. Indices don't have any inherent knowledge about the data they contain, even though it has a timestamp field and a name that reflects some dates, Elasticsearch doesn't know if that index contains data that falls in that 24 hour period without looking.

I am curious where you are seeing it query the logs-2016.10.[1-30] pattern as well. Are you watching the Elasticsearch logs or do you see Kibana making a number of requests? Kibana does use the field stats API to determine which indices contain records in a given data range and only query those, so now i'm curious if perhaps that isn't working as expected with index aliases, or perhaps there's some other kind of bug there.

5

  1. Yes you are correct, I have an alias name logs pointing to logs-2016.10.[1-30] and I have simply used logs as the patten in Kibana.

  2. On kibana, it just queries against the alias with the start and end timestamp in millis which corresponds correctly to the filter I select lets say 24 hour.

  3. I see the different indexes query in the ES logs. Its querying indexes which it shouldn't.

6

Ok, so it sounds like it's probably not a Kibana issue then, at least for now :wink:

When you say it's "querying indexes which it shouldn't", do you mean it's querying beyond the week's worth of indices you defined in the alias, or simply that it's querying all 7 indices even though the data you are trying to look at only exists in 1, or at most 2, since you're limiting it to 24 hours?

7

You are right, its query all 7 indices even if it should look at 1 or at most 2 indices

8

OK. I believe this is simply how aliases work. Since your alias is actually a collection of indices, Elasticsearch has to look inside each one to determine if there are any documents that match the criteria of your search.

That is, you've asked it for data in the last 24 hours, but it doesn't know what data might be included as part of that collection of indices without looking in each one.

I believe the alias functionality in Elasticsearch is simply a way to control how data is read and indexed, as well as being a handy shortcut for querying multiple indices while specifying a single index. I don't personally have a lot of experience with them though, so there may be a way to give your alias some context about what data lives in which underlying indices, or otherwise restrict which indices it looks in, but I don't know of a way, and I don't see anything in the docs about that. You might try asking about it over in the Elasticsearch section, they'd certainly be more knowledgeable about aliases or other solutions to what you are trying to do than I am.

I hope that explanation about why it works that way at least makes sense though.

9

Thank you !! Appreciate it.
What you said makes sense.

For additional clarity , I posted the question in ElasticSearch forums as well.

10

You can edit the thread (the pencil icon near the title) and move this to Elasticsearch, it'll help maintain context for anyone else that reads things.

11

Thanks . Done !!