Logstah reading from elastic daily based index name

Elastic Stack Logstash
1

Hello,

I am new to logstash.I
I want to get data out of elasticsearch and filter it and send it to some third party sw.
With help of examples and docs the first draft seems to work.

But the elasticsearch index name is based on current date, so I am asking, how can I do this name more adoptable, every day current index has different name.

Here is example od my logstash.conf file:

It runs elasticsearch query for data newer than 1 minute and does so every minute.

input {
  elasticsearch {
hosts => "elasticsearch:9200"
index => "application-2020.12.01"
query => '{ "query": { "range": { "timestamp": { "gt": "now-1m"  }}}}'
schedule => "*/1 * * * *"
}
}

Of course next day logstash will need to read from index named application-2020.12.02 and so on. Only way to achieve this as I see is to create (by script) every day new logstash.conf file and then daily in crontab script newly start logstash process.
Is there any better way , like to define in the logstash.conf the changing name of index, that can actively change ?

thanks

2

You do not necessarily have to update the logstash.conf, you could reference an environment variable in the index option. You would still have to restart daily though.

Another possibility is to use index => "application-*". I have not run elasticsearch for a couple of years but if I recall correctly running a time based query against an index that returns no results is really cheap. Run both queries against elasticsearch and take a look at the took value in the response. You may find that telling it which index is for today is an optimization that simply is not worth it.

1 Like
3

Correct :slight_smile:

1 Like
4

Thanks a lot for the quick and really useful hints!

Solution using environment variable works very good.

The other idea I hop I understand it correctly.I dont think if it will work so good because
in my elasticsearch "application-*" will return like 10 indices, which are all full od data.

Regarding the querying of elasticsearch. As you can see from conf file, I run every minute elastic query for data newer than 1 minute. For now it works OK. Is there a more "ellegant" solution, like unix "tail", that would instantly(without 1 min delay) send data to logstas as soon new record is written to elastic index.

5

As I said, you should test it. I think you will be surprised how little it costs to include all the indexes.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.