Filebeat Index Template from Modules

[tactics]

Hi,

Not sure if I am overcomplicating things and bothering unnecessarily...

I have finally got apache logs importing nicely using Filebeat and the Apache module.

I am simply using the fields.yml and therefore the index template is massive... Is there any way to only include the fields that are used in the module or maybe use the Index to generate fields and mappings that are being used?

Maybe I don't actually need to worry about this as the field definitions and mappings don't take up space or cause issues... The only downside I can see is when viewing data to make new searches etc there is a huge number of fields to scroll through!

Any help, advice or reassurance greatly appreciated.

[mtojek (Marcin Tojek)]

I'm not quite convinced if there is an easy workaround for you. You can review the Ingest Management feature in Kibana, which changes a bit the approach on how fields are defined and limit them only to required ones.

[tactics]

Thanks mtojek, I had looked at that! Trying another route as well so will report back if I manage to get anything working. 3.7GB of logs translated to a 6.3GB Elastic index, so not a complete nightmare but have a feeling this might be reduced with less data fields?

[tactics]

@mtojek - just tried copying the fields.yml and opening it in Sublime text, this allows me to collapse field groups:

I then removed the ones that weren't needed:

My original index template JSON file had 1,007 lines and the mappings had 16,172 lines. I saved the new fields as apache-fields.yml - updated this in filebeat.yml and then changed all of the index and template settings so a new index patten was used.

Ran filebeat setup and the new index template JSON file has 364 lines and the mappings has 3,242 lines.

Ran a test import on one log file - the Kibana [Filebeat Apache] Access and error logs ECS Dashboard works nicely still (change setup.dashboards.index: to reflect the new index names and updated templates) so seems to be fine.

Just running the same log files (14 million lines) into the system and will report back if there is a size difference or not. Should have recorded time on the first import as well to see if that changes as well but will probably run the imports again later on so can always check then.

[tjfred]

what was the actual problem you were facing by having all the fields present in fields.yml?

[tjfred]

fields in the template that are not used are not going to take up space in your elasticsearch index. perhaps you are suffering with this bug: Filebeat's default index mapping causes Kibana to display many empty date fields (now fixed)