Elasticsearch filebeat index name

[hoangvnit (Hoang Le)]

Hi

Currently i'm using filebeat & logstash forward nginx log to elasticsearch.
this is my config in logstash for output to elastic

output {
  elasticsearch {
    hosts => "https://search-member-recipes-i6pcn3evnwfokrka6mech2i3hq.us-west-2.es.amazonaws.com:443"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #index => "%{[@metadata][beat]}-log"
  }

i don't want use format %{+YYYY.MM.dd} for index, because when we have 365day, on elasticsearch will be have 365 index.

and when i changed to %{[@metadata][beat]}-log, then elasticsearch Visualize can't display nginx log info.

anyone know what problem here, please help me

thank andvances

[jsoriano (Jaime Soriano)]

In general, it is recommended to have more smaller indexes instead of a very big one, it uses to lead to better perfomance and less maintenance overheads. For example with one index per day you can easily delete old indexes without affecting recent data, or archive old data in slower and cheaper disks as it uses to be less frequently accessed, something very useful when storing time sensitive information as logs or metrics.

In any case, even if daily indexes are the default, other name patterns should work too, please check that you have an index pattern configured in Kibana that matches your index name, and that you select this index pattern in your searchs and visualizations.