How to stop logstash send data to mongo if data incorrect

Elastic Stack Logstash
1

Hi All,

It seems to a loop which cannot stop when logstash would like to write incorrect data to mongo, even I did 'service logstash restart', the logstash started and then start to try send the incorrect data

My data process. logstash > kafka > logstash > elasticsearch, mongo

grep logstash process, cannot kill 18528 and et al

logstash 18258 2.0 4.0 4081932 325384 pts/0 SNl 15:12 0:46 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
logstash 18365 2.6 3.8 4081948 311148 pts/0 SNl 15:23 0:43 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -...

logstash setting

input {
        kafka {
                topic_id => "neptunus"
                zk_connect => "zookeeper4kafka:2181"
                type => "test"
        }
}

output {
        stdout { codec => rubydebug }
        elasticsearch {
                host => "140.92.25.59"
                codec => "json"
                protocol => "http"
                index => "logstash-%{+YYYY.MM.dd}"
        }

        mongodb {
                uri => "mongodb://mongo4kafka"
                database => kafkatest
                collection => hellomongo
                codec => "json"
        }

        mongodb {
                uri => "mongodb://mongo264kafka"
                database => kafkatest
                collection => hellomongo
                codec => "json"
        }
}

Following message print continuously

{:timestamp=>"2015-10-15T15:38:26.293000+0800", :message=>"Failed to send event to MongoDB", :event=>#<LogStash::Event:0x24f2ce5c @metadata_accessors=#<LogStash::Util::Accessors:0x4e4fecf4 @store={"retry_count"=>0}, @lut={}>, @cancelled=false, @data={"message"=>"{a:22,"name":"franky"}", "@version"=>"1", "@timestamp"=>"2015-10-15T07:32:04.468Z", "host"=>"logstashpubkafka", "path"=>"/var/log/kafka/test", "type"=>"test", "tags"=><Java::JavaUtil::ArrayList:89942749 ["_jsonparsefailure"]>}, @metadata={"retry_count"=>0}, @accessors=#<LogStash::Util::Accessors:0x23f04235 @store={"message"=>"{a:22,"name":"franky"}", "@version"=>"1", "@timestamp"=>"2015-10-15T07:32:04.468Z", "host"=>"logstashpubkafka", "path"=>"/var/log/kafka/test", "type"=>"test", "tags"=><Java::JavaUtil::ArrayList:89942749 ["_jsonparsefailure"]>}, @lut={"type"=>[{"message"=>"{a:22,"name":"franky"}", "@version"=>"1", "@timestamp"=>"2015-10-15T07:32:04.468Z", "host"=>"logstashpubkafka", "path"=>"/var/log/kafka/test", "type"=>"test", "tags"=><Java::JavaUtil::ArrayList:89942749 ["_jsonparsefailure"]>}, "type"], "@timestamp"=>[{"message"=>"{a:22,"name":"franky"}", "@version"=>"1", "@timestamp"=>"2015-10-15T07:32:04.468Z", "host"=>"logstashpubkafka", "path"=>"/var/log/kafka/test", "type"=>"test", "tags"=><Java::JavaUtil::ArrayList:89942749 ["_jsonparsefailure"]>}, "@timestamp"]}>>, :exception=>#<NoMethodError: undefined method bson_type' for <Java::JavaUtil::ArrayList:89942749 ["_jsonparsefailure"]>:Java::JavaUtil::ArrayList>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.6-java/lib/bson/hash.rb:44:in to_bson'", "org/jruby/RubyHash.java:1341:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.6-java/lib/bson/hash.rb:43:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.6-java/lib/bson/encodable.rb:57:in encode_with_placeholder_and_null'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.6-java/lib/bson/hash.rb:42:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.6-java/lib/bson/array.rb:49:in to_bson'", "org/jruby/RubyArray.java:1613:in each'", "org/jruby/RubyEnumerable.java:978:in each_with_index'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.6-java/lib/bson/array.rb:46:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.6-java/lib/bson/encodable.rb:57:in `encode_with_placeholder_and_null'",

2

What do your mongo logs say about this?

3

Hi Mark,

seems to if the log data parse to json failure, and the sent data include "["_jsonparsefailure"]"

then I don't know why logstash always run not only one proecess, and I cannot kill the process id btw

logstash  1606  0.3  4.3 4084448 351232 ?      SNl  Oct15   5:11 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
logstash  6747  4.4  3.9 4081488 319656 pts/0  SNl  14:07   0:40 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
logstash  6851 11.1  3.8 4079440 310936 pts/0  SNl  14:17   0:31 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
4

What Logstash version?

5

Hi Mark,

logstash-1.5.2-1.noarch.rpm

6

The logstash process seems to open another process by itself if there are some errors occurs (e. g. like cannot send the message to the destination) ?

Anyone have idea?

Jason