Server based filebeat index name

Elastic Stack Beats
1

Hi,

If filebeat is installed in multiple servers. How to make sure that that logstash able to make index in elasticsearch like filebeat-{servername}-*.

As of now default one is filebeat-*. The version i am using of filebeat, ELK is same 6.4.

2

Hi @Chetan_Rana,

You can change the index name to add below configuration in your filebeat.yml file.

Optional index name. The default is "filebeat" plus date and generates [filebeat-]YYYY.MM.DD keys.

index: "filebeat-%{+yyyy.MM.dd}"

Regards,
Harsh Bajaj

3

Hi @harshbajaj16 Thanks for reply. Do you mean i need to place index like this in filebeat.yml

output.logstash:

The Logstash hosts

hosts: ["x.x.x.x:5044"]

index: "filebeat-servername-%{+yyyy.MM.dd}"

As i have filebeat installed in multiple server so need its specific name to appear in index.

Regards
Chetan

4

Is there a particular reason why you need different indexes for each server? Maybe you need to separate them for security access or some other reason? I ask because if there is no real reason then it's not typically necessary and will just create more shards in your cluster that may be unnecessary (generally speaking...many different use cases and configurations).

5

Hi @bigphil

Yes there is a reason in doing so as i have 50+ servers whose logs are required to collected in elasticsearch.
So if the index name have specific server name in it, this shall make easier for me to see which index data is getting used in discover page and making dashboards in kibana I believe.

Let me know if i am on right track.

Regards
Chetan

6

Hi @Chetan_Rana,

Yes, you need to specify the same in all filebeat node in filebeat.yml file.

And replace "servername" with valid environment variable to get the actual servername otherwize it will print as "servername" keyword in index name.

for E.g $HOSTNAME

Regards,
Harsh Bajaj

7

If thats the case, it's not really necessary to create a per server index. You should be able to easily use filters and queries using Kibana Discover and use saved searches that target a specific server when creating Visualizations and Dashboards so you only look at specific servers events. Better yet would be to use the Kibana visualization "Control" type and then you could just pick the server events you want to see when its part of a dashboard. Filebeat sends a couple of fields already with the servers hostname, i.e. beat.hostname and host.name, that you could use. I'd then keep an eye on the shard size and if the amount of data is becoming to large for a single shard, use the rollover API or on in your Filebeat config, change the number of shards. This blog post about shard/shard sizing may be a good read for you.

8

@bigphil Thanks for your inputs ... it really helped

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.