Logstash save/ modify configuration in environment

[Manoj_Hettiarachchi]

Hello,

In my system, I use logstash, filebeat and elasticsearch
Filebeat reads the logs, required fields in the logs are filtered with logstash and saved in elasticsearch.

I have a customer requirement to switch on/off saving some fields in the log by a single config change by the customer.

My planned approach is to keep the switch variable as an environment variable in "/etc/default/logstash" location and let the customer change the variables with a file operation.

But I have found out that the logtash config is not reloaded when we change that file even if we set the "config.reload.automatic: true". So I cannot continue my planned approach.
Also letting customer edit the logstast ".conf" files is not a good approach either because the code is so complex.

Please advice on this issue.

Thanks,

[Manoj_Hettiarachchi]

I have found a alternative approach to complete this requirement.
Instead of using the environment variables I have used a text file to save my switch variables.
Then used a ruby code to read file and add the variable values to the event.

ruby {
code => "event.set( 'variable1',IO.readlines('/etc/logstash/input.txt')[0])"
}

This has fixed my problem. But I would like to know is there a performance impact in executing file operation in each event

[Manoj_Hettiarachchi]

In my above solution, logstash will have to read the file in every event, so I have changed my config to following. So logstash will read form a JSON file and keep it in memory.

ruby {
      init => "
      require 'json'
        file = File.read('/etc/logstash/config.json') 
        config_object = JSON.parse(file)
        @api = config_object['api']
        @request = config_object['request']
        @response = config_object['response']"
      code =>"
        event.set('show_api', @api)
        event.set('show_request', @request)
        event.set('show_response', @response)"
    }

Thanks