Disable index rotation in logstash

(Hayder Abbass) #1


By default, logstash create indices in the format logstash-YYYY.MM.DD, which essentially creates one index per day. Is it possible to disable this feature and force logstash to push data to a specific index name?

Thanks for your help :slight_smile:

(Mark Walkom) #2

Yes, just change the index name in the output.

But my question would be; Why do you want to do this?

(Hayder Abbass) #3

Thanks for your help Mark. I had like to push all log data in one index per customer. We will normally have a maximum of 5 million records per index. Do you think it is a good idea? (I'm a total noob in ElasticSearch).

(Mark Walkom) #4

No, because how are you going to age data out easily?

You're better off using time based indices!

(Hayder Abbass) #5

Thats an interesting point. Actually, we have to retain logs for 2 years. I'm not sure if rotating indexes every day is a good idea as we are going to have hundreds of indices per customer. Do you think it is a good idea? Also, is there a performance gain if we store indexes by day?

(Mark Walkom) #6

Move to weekly/monthly instead of daily.

(system) #7