Disable index rotation in logstash


(Hayder Abbass) #1

Hello,

By default, logstash create indices in the format logstash-YYYY.MM.DD, which essentially creates one index per day. Is it possible to disable this feature and force logstash to push data to a specific index name?

Thanks for your help :slight_smile:


(Mark Walkom) #2

Yes, just change the index name in the output.

But my question would be; Why do you want to do this?


(Hayder Abbass) #3

Thanks for your help Mark. I had like to push all log data in one index per customer. We will normally have a maximum of 5 million records per index. Do you think it is a good idea? (I'm a total noob in ElasticSearch).


(Mark Walkom) #4

No, because how are you going to age data out easily?

You're better off using time based indices!


(Hayder Abbass) #5

Thats an interesting point. Actually, we have to retain logs for 2 years. I'm not sure if rotating indexes every day is a good idea as we are going to have hundreds of indices per customer. Do you think it is a good idea? Also, is there a performance gain if we store indexes by day?


(Mark Walkom) #6

Move to weekly/monthly instead of daily.


(system) #7