Indice names

[hueyg]

Have a new ELK installation based on online tutorials which is currently scraping a log sent from a SINGLE generic syslog server. I am new so still very green on how many of the pieces fit together. RTFM is not working for me.

In Kibana you create an index based off of a discovered indice that is being sent to it from Logstash in my case. This indice has a name of filebeat as the syntax based on my installation. Everything works with a single server but need some clarification when I expand to multiple servers.

1. Where is the actual naming syntax specified in an ELK stack? At the Filebeat or Logstash level? Can I get pointed to the actual line in whatever .conf file this is usually specified?

2. Say I have two servers and leave both of them configured to send to the same indice name. On the same day does Elasticsearch for Logstash then merge the two "streams" or will I see two separate indices with the same name in Kibana?

3. How does everyone else handle multiple servers? Different indice for every server, group them by application (syslog, apache, etc...) or just let them exist in one single indice and rely on elasticsearch to sort through it after the fact.

Hope the questions make sense and thanks for everyone's time.

[Christian_Dahlqvist]

@hueyg

Where is the actual naming syntax specified in an ELK stack? At the Filebeat or Logstash level? Can I get pointed to the actual line in whatever .conf file this is usually specified?

The index name is specified in the elasticsearch output plugin, and the index is created when data is written to it from logstash. You can override this using the index setting, but be aware this mean the default index template that matches the default index name will no longer apply.

@hueyg

Say I have two servers and leave both of them configured to send to the same indice name. On the same day does Elasticsearch for Logstash then merge the two "streams" or will I see two separate indices with the same name in Kibana?

No, data from multiple sources are generally stored in the same indices.

@hueyg

How does everyone else handle multiple servers? Different indice for every server, group them by application (syslog, apache, etc...) or just let them exist in one single indice and rely on elasticsearch to sort through it after the fact.

Often the type of data rather than the source determines the index to put it in. Similar types of data is generally stored in the same index. Creating lots of very small indices and shards can cause a lot of problems as there is overhead associated with each shard that is created. Please read this blog post for guidance on shards and sharding.

[hueyg]

Thanks for the clarification Christian. Do you have any advice for how to track down the problem when one server is not sending data if multiple servers are dumping into one indice?