# LS 1.5.1: Split json including arrays

**URL:** <https://discuss.elastic.co/t/ls-1-5-1-split-json-including-arrays/24449>\
**Category:** Logstash\
**Created:** [June 26, 2015, 5:41pm UTC](https://discuss.elastic.co/t/ls-1-5-1-split-json-including-arrays/24449 "2015-06-26T17:41:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![gefa](https://avatars.discourse-cdn.com/v4/letter/g/a88e4f/32.png) [@gefa](https://discuss.elastic.co/u/gefa)\
**Post date:** [June 26, 2015, 5:41pm UTC](https://discuss.elastic.co/t/ls-1-5-1-split-json-including-arrays/24449/1 "2015-06-26T17:41:46Z")

</div>

Hi all,

My used software:

- Debian 8.1
- logstash 1:1.5.1-1, installed via the provided .deb
- java version "1.8.0\_45" from the Oracle website

What I would like to achieve:  
I've got some logs in json format which includes arrays. One example:

```
{
"id": "9bw4rye7h38hgg5k",
"events": [
    {
        "ts": "2015-06-26T16:52:18.628Z",
        "evt": "details:close",
        "data": {
            "entry": 774609
        }
    },
    {
        "ts": "2015-06-26T16:52:18.628Z",
        "evt": "details:display",
        "data": {
            "entry": 774188
        }
    }
]
}

```

Another example:

```
{
"id": "iri4dy894v2l2xde",
"events": [
    {
        "ts": "2015-06-26T00:00:00.073Z",
        "evt": "details:display",
        "data": {
            "entry": 773249
        }
    }
]
}

```

[So, as you see, sometimes there are multiple events entries, sometimes only one. Is it even possible to store all these details in elasticsearch using logstash?]

What I did so far:

- Using the code provided at [1]. After restarting logstash, it seems that no processing is happening, at least no data gets written to elasticsearch. Setting the log to debug gives me [2].
- Using the json and split filters, like

> json {  
> source =\> "json\_raw"  
> }  
> split {  
> field =\> "events"  
> }

worked once, directly after restarting logstash. After one nearly sucessfull (the data in question included two event entries, one got stored [3]) processed and stored log entry, it seems that logstash again "stopped" somehow: The daemon is still running, however, no data gets written to elasticsearch anymore; I wasn't able to find anything suspicious in the log, besides [4] [?].

I've googled all day long around how to handle this, but didn't found a solution.  
Any hints?

Help would be greatly appreciated!

Thanks and all the best,  
gefa

[1] [http://kapaski.github.io/blog/2014/07/24/logstash-to-parse-json-with-json-arrays-in-values/](http://kapaski.github.io/blog/2014/07/24/logstash-to-parse-json-with-json-arrays-in-values/)

[2]

> {:timestamp=\>"2015-03-10T18:06:26.946000+0100", :message=\>"Failed to flush outgoing items", :outgoing\_count=\>10, :exception=\>#\<Errno::EBADF: Bad file descriptor - Bad file descriptor\>, :backtrace=\>["org/jruby/RubyIO.java:2097:in `close'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb:173:in `connect'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb:139:in `connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:406:in `connect'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/pool.rb:48:in `fetch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:403:in `connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:319:in `execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:217:in `post!'", "/opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:106:in `bulk_ftw'", "/opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:80:in `bulk'", "/opt/logstash/lib/logstash/outputs/elasticsearch.rb:315:in `flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219:in `buffer\_flush'", "org/jruby/RubyHash.java:1339:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216:in `buffer\_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:193:in `buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:112:in `buffer\_initialize'", "org/jruby/RubyKernel.java:1521:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:110:in `buffer\_initialize'"], :level=\>:warn}

> {:timestamp=\>"2015-06-26T18:32:49.369000+0200", :message=\>"Exception in filterworker", "exception"=\>#\<LogStash::ConfigurationError: Only String and Array types are splittable. field:json\_raw is of type  
> = NilClass\>, "backtrace"=\>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-split-0.1.6/lib/logstash/filters/split.rb:46:in `filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/filters/base.rb:162:in `multi\_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/filters/base.rb:159:in `multi\_filter'", "(eval):142:in `cond_func_1'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):138:in `cond_func_1'", "(eval):126:in `filter\_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:218:in `filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:156:in `start\_filters'"], :level=\>:error, :file=\>"logstash/pipeline.rb", :line=\>"230", :method=\>"filterworker"}{:timestamp=\>"2015-06-26T18:37:01.730000+0200", :message=\>"Exception in filterworker", "exception"=\>#\<NameError: uninitialized constant LogStash::Filters::Ruby::JSON\>, "backtrace"=\>["org/jruby/RubyModule.java:2733:in `const_missing'", "(ruby filter init):4:in `parse\_json'", "(ruby filter code):1:in `register'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-0.1.5/lib/logstash/filters/ruby.rb:37:in `filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/filters/base.rb:162:in `multi\_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/filters/base.rb:159:in `multi\_filter'", "(eval):142:in `cond_func_1'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):138:in `cond_func_1'", "(eval):126:in `filter\_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:218:in `filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:1 56:in `start\_filters'"], :level=\>:error, :file=\>"logstash/pipeline.rb", :line=\>"230", :method=\>"filterworker"}

[3] Excerpt of the whole data stored in elasticsearch:

```
"json_raw": "{\"id\":\"9bw4rye7h38hgg5k\",\"events\":[{\"ts\":\"2015-06-26T16:52:18.628Z\",\"evt\":\"details:close\",\"data\":{\"entry\":774609}},{\"ts\":\"2015-06-26T16:52:18.628Z\",\"evt\":\"details:display\",\"data\":{\"entry\":774188}}]}",
"log_timestamp": "2015-06-26 18:52:18,679",
"id": "9bw4rye7h38hgg5k",
"events": {
  "ts": "2015-06-26T16:52:18.628Z",
  "evt": "details:close",
  "data": {
    "entry": 774609
  }
},

```

[4]

```
"tags"=>#<Java::JavaUtil::ArrayList:0x2a427100>

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:36am UTC](https://discuss.elastic.co/t/ls-1-5-1-split-json-including-arrays/24449/2 "2017-07-06T05:36:17Z")

</div>