Logstash not indexing the input data to elastic search

Elastic Stack Logstash
1

Hi,
I am trying to feed json array to elastic search. Its not throwing any error and not indexing the data to elastic search. I have given my config below. Please let me know if am making any mistake in my config.

input {
file {
path => "/Users/me/practice/logstash-2.4.0/stash/*.json"
codec => "json"
start_position => "beginning"
}
}
filter{
ruby {
code => "event['@metadata']['_id'] = event['_id']"
}
mutate{
remove_field => ["_id","path","host"]
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "logs"
document_id => "%{[@metadata][_id]}"
}
}

2

Please check out the Logstash posts from the past couple of days. I recall at least two people asking the same question.

What I can add is: Do not use the elasticsearch output until you've verified that your input files are being correctly read and processed. Use a stdout { codec => rubydebug } output instead.

3

Thank you for the response. I checked adding stdout. I don't see anything there. I can assume then its a problem with input plugin or its config. However when I add a specific json file. Its processing content of that particular file only. Any thoughts?

4

I checked the logs in debug mode. It prints the following repeatedly without processing the file content.

file grew: /Users/satheeshravi/practice/logstash-2.4.0/stash/my.json: old size 0, new size 994 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"254", :method=>"each"}
each: file grew: /Users/satheeshravi/practice/logstash-2.4.0/stash/test1.json: old size 0, new size 994 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"254", :method=>"each"}
each: file grew: /Users/satheeshravi/practice/logstash-2.4.0/stash/test1476119896.json: old size 0, new size 42459 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"254", :method=>"each"}

5

Created an issue for file input plugin