ISSUE
Adds all available datetime fields from index template with empty value (even those fields those are otherwise used by disabled modules and disabled) , when add date_index_name processor in pipeline of filebeat fortinet module.
Elasticsearch version : 7.8.0
Kibana version : 7.8.0
Filebeat version : 7.8.0
JVM version :
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1ubuntu1-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
OS version :
Linux elastic 5.4.0-40-generic #44-Ubuntu SMP Tue Jun 23 00:01:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
I wanted to create the dyname index name for fortinet logs using the date from the log entry. So in the pipeline.yml for the fortinet module, I added following processor.
- date_index_name:
index_name_prefix: "filebeat-7.8.0-firewall-"
field: fortinet.firewall.date
date_formats:
- "yyyy-MM-dd"
date_rounding: "d"
This does create the proper index name using the date value from the firewall log entry. But when the log data is processed with this updated pipeline, for each log record doc, it adds all available data-time fields from the respective index template (that are even of other modules in filebeat which are disabled) with empty value. And if we don't have this processor added in the pipeline, it works okay except that I can have dynamic index name using the date from the log record.
For your information, I have only Fortinet module enable for the filebeat.
Steps to reproduce:
- Install and configure the filebeat and enable fortinet module.
- Configured fortinet module to use "file" input for the syslogs of Fortinet logs.
- Run the filebeat module without and changes in the pipeline with the logs being feed to it.
- Logs are processed and the index docs are created with only required field for the Fortinet module.
- Now delete the indexes, to reprocess logs with update pipeline for fortinet module.
- Update the injest/pipeline.yml for fortinet.firewall module to add the above mention processor code after the kv processor.
- Setup this update pipeline using the "filebeat setup --pipelines" command.
- Clean the file registry to reprocess the fortinet logs.
- This time, it adds all available datetime fields from index template (even ones those are from disabled modules) with empty value which are not even being process by fortinet pipelines.
Please guide me on the same, if I am doing anything wrong. I don't want fields from other modules (even ones those are disable) in docs of indexes for the fortinet logs.