Logstash - multiple indices for one input file


(Krishna Chaitanya) #1

I want to create multiple indices using Logstash configuration into ES. From my understanding, it is easy to use type in file-input if we have multiple inputs. Then, create one index per one type.

But, I do not have multiple input files here. I have only one input-file. I just want to parse results differently. That means, for one index, I want to apply lot of filters(date,split,ruby..); for the other index, I do not want any filters applied. Finally put these indices into same elasticsearch output cluster.

How to achieve this?
Below is config file which describes what I want to achieve.

input {
  file {
    codec => json
    path => "/path/to/json"
    start_position => "beginning"
  }
}

filter {
   #### **apply these filters only for index2**
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "index1" 
  }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "index2" 
  }
  stdout { codec => rubydebug }
}

(Mark Walkom) #2

Use https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html and a bunch of conditionals :slight_smile:


(Krishna Chaitanya) #3

Thanks for the idea. I think that is going to work. Let me try


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.