Custom log file configuration

[Vijayakumar_Kannan (vijay kannan)]

i have the custom log file which i want to parse and send the logs to ES from the filebeat configuration. So i had the following

• filebeat configuration

filebeat.prospectors:
- type: log
  enabled: true
    - /tmp/custom.log
  pipeline: filebeat-custom-pipeline

• ES configuration

1. validated the pipeline with _ingest/pipeline/_simulate
2. posted the ingest pipeline in ES configuration

Now i had to change one field "clientip" type to "geopoint" data type. the article refers in filebeat go for new filebeat module development.

if i you can guide me for alternate way for creating the mapping for fields created out of custom file that would be more helpful.

[steffens (Steffen Siering)]

the article refers in filebeat go for new filebeat module development.

Which article?

You need to adapt the template mapping. This is normally done by adjusting fields.yml, but is also possible via json. There is a number of settings for template setup documented here. The setup.template.append_fields setting might fit your needs.

[Vijayakumar_Kannan (vijay kannan)]

thanks steffens, since i am using the filebeat 6.3.2 version so i had to use the fields.yml for field configuration and i am able to get it after the current index deletion and restarted the filebeat configuration.

i have a question

1. When i update the fields.yml does it take effect only on new index creation or any steps to be performed ?
2. if i have 500 servers and i need the custom fields.yml in single server (custom logs) in that server i had to enable the option of template overwrite?

[steffens (Steffen Siering)]

When i update the fields.yml does it take effect only on new index creation or any steps to be performed ?

The fields.yml is used to install a template mapping. This is a one time setup step. Afterwards Elasticsearch will use the template mapping to create an index.
You can use setup or enable template overwriting to force the template to be replaced. This will only affect new indices though.

if i have 500 servers and i need the custom fields.yml in single server (custom logs) in that server i had to enable the option of template overwrite?

Yes, overwrite might help here. But once you add another server needing to add another field you might have a problem. Then you should consider to have an extra centralized configuration with a centralized fields.yml and run run filebeat setup if required.