How to separate index of filebeat coming from 2 or more hosts

[mark.quilates (Mr.M)]

Hi Guys,

Can you help me, I have 2 filebeats in separate host and I used logstash pipeline.

The thing is I want the other filebeat it to stored its data in new index.

How can I make that? To have new index name in my elasticsearch?

Currently I have this filebeat-* index in my elasticsearch that used of two filebeat clients.

What I want is the new filebeat with new index name like this webpage-

[steffens (Steffen Siering)]

You can configure the elasticsearch output in logstash to use an event field.

E.g. add a field in filebeat identifying the kind of source via:

fields.application: webpage
fields_under_root: true

Having an application field from each filebeat you can configure the index name to be index => "%{[application]}-%{[@metadata][version]}-%{+yyyy.MM.dd}" (this should create an daily index with the beats version in the index name). Then in kibana you can create index patterns filebeat-* and webpage-*.

[mark.quilates (Mr.M)]

Hi @steffens,

Tried the above and changed my index output based on the given code but, it didn't display the correct index name. Please see below.

No webpage- or filebeat- index.

[Tek_Chand]

@Mark, Please try the below format in output filter in logstash to create the index:

"application-%{[@metadata][version]}-%{+YYYY.MM.dd}"

Thanks.

[steffens (Steffen Siering)]

The settings/names are for use with Filebeat and Logstash. Logstash did create an index named %{[application]}, because some event was missing the application field. You must ensure all events have an application field.

[mark.quilates (Mr.M)]

Hi @steffens and @Tek_Chand,

I added these:

fields.application: webpage
fields_under_root: true

to my 1st filebaet

these:

fields.application: filebeat
fields_under_root: true

to my 2nd filebeat

and these:

fields.application: metricbeat
fields_under_root: true

to my metricbeat

config but, still the index not displaying the expected name. Please see screenshot

Here's my logstash code: When I tried the given two codes above.

output {
elasticsearch {
hosts => [ "192.168.213.128:9200" ]
manage_template => false
index => "application-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}

output {
elasticsearch {
hosts => [ "192.168.213.128:9200" ]
manage_template => false
index => "%{[application]}n-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}

[Tek_Chand]

@mark, Are you want create 3 separate index for 3 different servers for filebeat?

Are you using filebeat as well as metricbeat?

Please provide above info so we can suggest solution accordingly.

Thanks.

[mark.quilates (Mr.M)]

@Tek_Chand

Yes, your are right. I have 2 filebeats and 1 metricbeat from 3 different servers.

[Tek_Chand]

@Mark, As per my knowledge for metricbeat you should use single index i.e metricbeat-* for all servers. Because its contain metrics of your servers and you can use metricbeat dashboard to visualize the data.

For filebeat you can create separate indexes on the basis of your data like for syslog, nginx log, auth log etc. You can create separate indexes. But filebeat dashboard are tied with filebeat-* index pattern. If you want to use filebeat dashboard for syslog, ssh log or nginx log these logs should be store in filebeat index.

To create separate index you need to handle it at Filebeat level and output fileter in logstash.

In filebeat you can use type for each logs type and use that type field in logstash output filter to create separate index for that log.

Refer the below examples:
filebeat.yml

- type: log
  paths:
    - /var/apps/shared/log/production.log
  fields_under_root: true
  fields:
    type: application_log


- type: log
  enabled: true
  paths:
     - /var/log/auth.log
  fields_under_root: true
  fields:
    type: auth_logs

Output Filter in logstash:

output {
  if [type] == "application_log" 
 {
  elasticsearch {
    hosts => ["xyz:9200"]
    sniffing => true
    manage_template => false
    index => "application-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
else
  {
elasticsearch {
    hosts => ["xyz:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
}
}
}

You can use nested if else also in output if you have define more than two fields.

Hope so it will help you. If you have any query please let me know.

Thanks.

[mark.quilates (Mr.M)]

@Tek_Chand

output { if [type] == "application_log" { elasticsearch { hosts => ["xyz:9200"] sniffing => true manage_template => false index => "application-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } else { elasticsearch { hosts => ["xyz:9200"] sniffing => true manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } }

@Tek_Chand,

Thank you very much for your time and efforts to help me. Finally it works now.

[Tek_Chand]

@Mark, Glad to hear that