Logstash ingest pipeline at elasticsearch

[Vijayakumar_Kannan (vijay kannan)]

Hi All

i m trying to use the filebeat -> logstash -> elasticsearch flow. When i enable the system module from filebeat the message doesn't parse so trying to use the output.elasticsearch pipeline option is not working.

ERROR

[2018-08-28T12:03:58,390][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"fileb
eat-6.2.4-2018.08.28", :_type=>"doc", :_routing=>nil, :pipeline=>"filebeat-6.2.4-system-syslog-pipeline"}, #<LogStash::Event:0xd3ae967>], :response=>{"index"=>{"_index"=>"filebeat-6.2.4-2018.08.
28", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"pipeline with id [filebeat-6.2.4-system-syslog-pipeline] does not exist"}}}}

Configuration

input {
  beats {
    port => 5044
   
  }
}
output {
  elasticsearch {
   hosts => "http://ese0001:9200"
   index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
   pipeline => "%{filebeat-6.2.4-system-syslog-pipeline}"
 }
}

[_Alex (Alex Stiff)]

Index and pipeline are both expecting strings. Here when you say "%{filebeat-6.2.4-system-syslog-pipeline}" this is looking for a variable called filebeat-6.2.4-system-syslog-pipeline which doesn't exist. If you remove the percentage signs from everything that's not a variable you should be OK.

output {
  elasticsearch {
   hosts => "http://ese0001:9200"
   index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
   pipeline => "filebeat-6.2.4-system-syslog-pipeline"
 }
}