Create dictionary file using logstash file output


#1

Hi everyone.
I have a scheduled input which pulls API and recieves batch of device details and refreshes device info in Elastcisearch index (so if I have 100 devices, i will always have 100 docs in elastic index). This input includes "device_id" and "device_hostname" fields.
I need to make dictionary for translate filter from this values, in form: "device_id: device_hostname".
My output looks like this;

  output {
   file {
     path => "/tmp/dict_hosts"
     codec => line { format => "%{device_id} : %{device_hostname}"} 
   } 
 }

The problem is that every time output executes it adds the same 100 "device_id: device_hostname" pairs to the end of file and it grows. If I use "write_behavior => overwrite" than I get only one line in dictionary.
Is there a way to overwrite file once per batch write?


(Faulander) #2

Since you only have two options for the file output which you already tried, i would create a small batch or python script which deletes the file before writing to it from logstash.

Maybe this link gets you going: