Hi all
I'll try to summarize our issue we're having here a bit to share as much information as possible. We're running a logstash architecture Shipper -> Redis-Queue -> Indexer ( -> graphite output) which runs fine with Logstash 1.4 (for some months now, in production).
We started testing Logstash 2.1 in our testing environment and struggled with an issue on processing events (at indexer). We could narrow down the issue that Logstash 1.4.x and 1.5.4 (including) do process events without issues on the indexer node (shipper remains on 1.4). Starting with 1.5.5 (and including 2.1.0) we're seeing exceptions and are missing events in graphite (it doesent matter if we update shipper or indexer or both). The message Logstash 1.5.5 is showing (while running with "--debug" and simplified input / output config, no filters in use) is:
...
:class=>"Java::JavaLang::ArrayIndexOutOfBoundsException",
:backtrace=>["com.fasterxml.jackson.core.sym.ByteQuadsCanonicalizer.addName(ByteQuadsCanonicalizer.java:853)",
"com.fasterxml.jackson.core.json.UTF8StreamJsonParser.addName(UTF8StreamJsonParser.java:2340)",
"com.fasterxml.jackson.core.json.UTF8StreamJsonParser.findName(UTF8StreamJsonParser.java:2224)",
"com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseLongName(UTF8StreamJsonParser.java:1831)",
"com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName2(UTF8StreamJsonParser.java:1786)",
"com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName(UTF8StreamJsonParser.java:1743)",
"com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1678)",
"com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:740)",
...
"org.jruby.RubyProc.call(RubyProc.java:228)",
"org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)",
"java.lang.Thread.run(Thread.java:745)"],
:level=>:warn,
:file=>"logstash/codecs/json.rb",
:line=>"58",
:method=>"decode"}
we're running CentOS 6.7 servers with Logstash packages from elastic.co repositores.
We did check the changelog (compared 1.5.4 to 1.5.5) and there are some hints / code changes which could be the cause for our failures. Unfortunately Logstash 1.5.5 does start processing some events (less than 100) before it struggles and logs the error and we were not able to revert changes from a 1.5.5 installation to successfully process all events.
Beside the changes in logstash core we saw code updates in the logstash-codec-json which is being updated from version 1.0.1 to 1.1.0 (from logstash 1.5.4 and 1.5.5).
Anyone could give us some hints to further identify the problem? Maybe anyone else had trouble with logstash release after 1.5.4 and redis/json events?
Thank you very much for any help!