Is it mandatory to load the index template into Elasticsearch manually when using filebeat?

[amruth]

Hi,

I am pretty new to Filebeat. I am using Logstash to send logs to another Logstash which will send logs to Elasticsearch(Logstash1 to Logstash2 to Elasticsearch). Now I am planning to replace Logstash1 with Filebeat. Documents say to load the index template into Elasticsearch manually and my question is - if I am already using a template in Elasticsearch, do I still need to load the index template into Elasticsearch manually?

Thanks

[andrewkroh]

It depends on where you are writing the Filebeat data to and whether the index template you have contains the appropriate mappings for the Filebeat fields. To be safe I would follow the directions.

My recommendation is to install the index template provided by Filebeat and write your data into the prescribed filebeat-<version>-* indices. Basically following the documentation and using the defaults. So you would manually load the Filebeat template as per the instructions. This template will apply to indices matching filebeat-6.2.4-*.

Then configure Logstash to output the beat data with this config.

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
    document_type => "%{[@metadata][type]}" 
  }
}

If you make any adjustments to index naming pattern then the template also needs to be changed.

[amruth]

Hi Andrew,

As per your suggestion, I would install the index template. But I have 4 Elasticsearch nodes(one coordinating node and 3 Data nodes). Do I need to install template on all 3 Data nodes one at a time using

curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_template/filebeat-6.2.4 -d@filebeat.template.json

And also is there a way to drop Beat fields? I am basically looking for same exact fields what I have right now with Logstash.

[andrewkroh]

The template becomes part of the cluster state so you can add it to any one node and it will sync to the others.

You can drop fields on the Filebeat side with a drop_fields processor.

processors:
 - drop_fields:
     fields: [field1, field2]

[amruth]

Thanks Adrew.

I have another question - I am ingesting logs to an index called smaple-%{+YYYY.MM} and I don't have template in elasticsearch and am planning to have one template by the time it creates next index(i.e next month).

Since I am planning to replace Logstash1 with Filebeat, I will have to load the template manually. How should I create a template in elasticsearch for next month?

I am confused between Filebeat template and the one we create in ES using Index Template

Thanks

[andrewkroh]

You can let Filebeat generate an index template for your version of Elasticsearch. Then you can manually install it. Between those steps you can customize the index template as needed (like customize number of shards or add additional fields).

# Export template to file.
filebeat export template --es.version=6.2.4 -E setup.template.pattern="smaple-*" > filebeat.template.json

# Load template to ES.
curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_template/smaple @filebeat.template.json