Output configuration question

Elastic Stack Logstash
1

Hi everyone,

I am curious about the code described here :

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" 
    document_type => "%{[@metadata][type]}" 
  }
}

I've run this config for a year now and this is creating a lot of indexes (one or more per day).
Is this supposed to be a code example to illustrate the possibilities offered in output, or the best way to organize the data ?
Should I have simply created a single index like this ?

index => "%{[@metadata][beat]}"

Thanks for your help!

2

How large are your daily indicies? This will be the determining factor. I would not remove the date element completely as eventually the indicies will become too large, and deleting data to make space will be a more complex process. However, you may be able to move from daily to monthly indicies if your current daily indicies are really small (e.g. < 1GB).

3

This is a good question, I think they may be <100mb / day.
Is there quick way to get this info (like average size) ?

4

You can access data about the indicies by querying the REST API. In Kibana go to Dev Tools -> Console and execute the query:

GET _cat/indices?v
5

It seems I was wrong about the size. My indices are between 200mb and 500mb / day

EDIT: I did a quick average on a year on my two outputs.
The first one is approx sending 20mb/day, but the second one is around 350mb/day.

I think i'll follow your input and change how the first is configured to something like

"%{[@metadata][beat]}-%{+YYYY.MM}"
6

That is pretty small. I if changed to monthly indices, by changing this...

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

to this...

index => "%{[@metadata][beat]}-%{+YYYY.MM}"

You will reduce the number of indices significantly. The only downside is that if you want to drop old data by simply deleting old indices, you would have to drop a full month at a time, but this is likely not an issue in your case.

7

Thanks for your help, i'll think about it.