LS 1.5.1: Split json including arrays

Elastic Stack Logstash
1

Hi all,

My used software:

  • Debian 8.1
  • logstash 1:1.5.1-1, installed via the provided .deb
  • java version "1.8.0_45" from the Oracle website

What I would like to achieve:
I've got some logs in json format which includes arrays. One example:

{
"id": "9bw4rye7h38hgg5k",
"events": [
    {
        "ts": "2015-06-26T16:52:18.628Z",
        "evt": "details:close",
        "data": {
            "entry": 774609
        }
    },
    {
        "ts": "2015-06-26T16:52:18.628Z",
        "evt": "details:display",
        "data": {
            "entry": 774188
        }
    }
]
}

Another example:

{
"id": "iri4dy894v2l2xde",
"events": [
    {
        "ts": "2015-06-26T00:00:00.073Z",
        "evt": "details:display",
        "data": {
            "entry": 773249
        }
    }
]
}

[So, as you see, sometimes there are multiple events entries, sometimes only one. Is it even possible to store all these details in elasticsearch using logstash?]

What I did so far:

  • Using the code provided at [1]. After restarting logstash, it seems that no processing is happening, at least no data gets written to elasticsearch. Setting the log to debug gives me [2].
  • Using the json and split filters, like

json {
source => "json_raw"
}
split {
field => "events"
}

worked once, directly after restarting logstash. After one nearly sucessfull (the data in question included two event entries, one got stored [3]) processed and stored log entry, it seems that logstash again "stopped" somehow: The daemon is still running, however, no data gets written to elasticsearch anymore; I wasn't able to find anything suspicious in the log, besides [4] [?].

I've googled all day long around how to handle this, but didn't found a solution.
Any hints?

Help would be greatly appreciated!

Thanks and all the best,
gefa

[1] http://kapaski.github.io/blog/2014/07/24/logstash-to-parse-json-with-json-arrays-in-values/

[2]

{:timestamp=>"2015-03-10T18:06:26.946000+0100", :message=>"Failed to flush outgoing items", :outgoing_count=>10, :exception=>#<Errno::EBADF: Bad file descriptor - Bad file descriptor>, :backtrace=>["org/jruby/RubyIO.java:2097:in close'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb:173:in connect'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb:139:in connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:406:in connect'", "org/jruby/RubyProc.java:271:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/pool.rb:48:in fetch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:403:in connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:319:in execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:217:in post!'", "/opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:106:in bulk_ftw'", "/opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:80:in bulk'", "/opt/logstash/lib/logstash/outputs/elasticsearch.rb:315:in flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219:in buffer_flush'", "org/jruby/RubyHash.java:1339:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216:in buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:193:in buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:112:in buffer_initialize'", "org/jruby/RubyKernel.java:1521:in loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:110:in buffer_initialize'"], :level=>:warn}

{:timestamp=>"2015-06-26T18:32:49.369000+0200", :message=>"Exception in filterworker", "exception"=>#<LogStash::ConfigurationError: Only String and Array types are splittable. field:json_raw is of type
= NilClass>, "backtrace"=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-split-0.1.6/lib/logstash/filters/split.rb:46:in filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/filters/base.rb:162:in multi_filter'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/filters/base.rb:159:in multi_filter'", "(eval):142:in cond_func_1'", "org/jruby/RubyArray.java:1613:in each'", "(eval):138:in cond_func_1'", "(eval):126:in filter_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:218:in filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:156:in start_filters'"], :level=>:error, :file=>"logstash/pipeline.rb", :line=>"230", :method=>"filterworker"}{:timestamp=>"2015-06-26T18:37:01.730000+0200", :message=>"Exception in filterworker", "exception"=>#<NameError: uninitialized constant LogStash::Filters::Ruby::JSON>, "backtrace"=>["org/jruby/RubyModule.java:2733:in const_missing'", "(ruby filter init):4:in parse_json'", "(ruby filter code):1:in register'", "org/jruby/RubyProc.java:271:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-0.1.5/lib/logstash/filters/ruby.rb:37:in filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/filters/base.rb:162:in multi_filter'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/filters/base.rb:159:in multi_filter'", "(eval):142:in cond_func_1'", "org/jruby/RubyArray.java:1613:in each'", "(eval):138:in cond_func_1'", "(eval):126:in filter_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:218:in filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:1 56:in start_filters'"], :level=>:error, :file=>"logstash/pipeline.rb", :line=>"230", :method=>"filterworker"}

[3] Excerpt of the whole data stored in elasticsearch:

"json_raw": "{\"id\":\"9bw4rye7h38hgg5k\",\"events\":[{\"ts\":\"2015-06-26T16:52:18.628Z\",\"evt\":\"details:close\",\"data\":{\"entry\":774609}},{\"ts\":\"2015-06-26T16:52:18.628Z\",\"evt\":\"details:display\",\"data\":{\"entry\":774188}}]}",
"log_timestamp": "2015-06-26 18:52:18,679",
"id": "9bw4rye7h38hgg5k",
"events": {
  "ts": "2015-06-26T16:52:18.628Z",
  "evt": "details:close",
  "data": {
    "entry": 774609
  }
},

[4]

"tags"=>#<Java::JavaUtil::ArrayList:0x2a427100>