Reading the properties file from input section of logstash

[kiranilla (Kiran)]

Hi,

I am able to read the properties file from filter section of logstash but not from input section.
Could you please help me out in reading the .properties file from input section of logstash.
Thanks in advance.

Regards,
Kiran.

[guyboertje]

You will need to give us more info. Please explain in detail. I don't know what you mean by properties file from the filter

[kiranilla (Kiran)]

HI @guyboertje,

Thanks for your reply.

Here is the example i worked on :
sample.properties

    AGSS,URO=http://localhost:8980/akn/
    AGCM,FTP,SMTP=http://localhost:8800/akn/gsk/se

1. I read the properties file from filter section of logstash file using ruby code: (which is possible)

   filter {
              csv {
            	   columns => ["Incident ID","Status","Resolved By","Resolution Breached","Resolution Month","Resolution Date","Resolution Analysis Code","Closure Code","Primary CI"]
   
    	 separator => ","
    	
      }
      
      		ruby {
    				code => '
    				         if event.get("Category") == nil 
    						 file = File.open("/C:\\ELK\\DataSet\\sample.properties", "r")
    						 data=file.read
    						 file.close
    						 dataarr=data.split("\n")
    						 dataarr.each do |da|
    							halfda=da.split("=")
    							subda=halfda[0].split(",")
    								subda.each do |sda| 
    										if event.get("Description").to_s.include?sda 
    							
    												event.set("Category",halfda[1])
    										end 
    								end
    							end
    							
    							end
    			
    						'
    				}
    }
   

2. In the above scenario i am able to read the properties file and getting the output, but the properties file is continuously read every time based on csv rows in a file (eg: csv file contains 100 rows the filter section will execute 100 times).

3) Instead of i would like to read properties file only once in the input section of the logstash and take the content of the properties file(ie.,URLs) in it and place them in the "URL" plugin in the input section of logstash.

something like below code of input section of logstash:

input {
http_poller {
urls => {
EL=> {
method => get
url => "http://localhost:8980/akn" # This url should be dynamic
verify_cert => false
headers => {
Accept => "application/json"
}
}
}

_url => "http://localhost:8980/akn_
The "url" should be dynamic (which need to be taken from the properties file)

I checked for logstash api where "ruby plugin" is available in the filter section but its not available in the input section.

It would be great if you can you please help me in this scenario,
Thanks in advance.

[guyboertje]

There is no ruby input - only a ruby filter. Plugins of type input are put in the input section of the config, similarly for filter.

You don't need to go to your proposed extremes.
You can use a translate filter and a json filter.
Write a little ruby/python script to extract and denormalise the sample.properties file into an expanded_properties.csv file.

AGSS,http://localhost:8980/akn/
URO,http://localhost:8980/akn/
AGCM,http://localhost:8800/akn/gsk/se
FTP,http://localhost:8800/akn/gsk/se
SMTP,http://localhost:8800/akn/gsk/se

I presume that there are a fixed number of URLs in this file so now your script can fetch the JSON from the URLs and replace the URL with the JSON body.

AGSS,{"foo": "bar"}
URO,{"foo": "bar"}
AGCM,{"baz": "bar"}
FTP,{"baz": "bar"}
SMTP,{"baz": "bar"}

Then using a translate filter you can enhance the event with the JSON. After the translate filter you use a JSON filter, it will unpack the JSON field and add the KV pairs.

https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html

Once you have this working you can experiment with not denormalising the properties file and writing the LHS as a regex instead see: https://stackoverflow.com/questions/41522280/how-to-use-regex-in-yaml-file-for-logstash-translate-filter.
This for your translate filter dictionary_path file contents maybe:

^(URO|AGSS)$,{"foo": "bar"}
^(AGCM|FTP|SMTP)$,{"baz": "bar"}