# Bug parsing json **URL:** \ **Category:** Logstash\ **Created:** [May 25, 2016, 8:38am UTC](https://discuss.elastic.co/t/bug-parsing-json/50921 "2016-05-25T08:38:59Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![JvonRudno](https://avatars.discourse-cdn.com/v4/letter/j/258eb7/32.png) [@JvonRudno](https://discuss.elastic.co/u/JvonRudno)\ **Post date:** [May 25, 2016, 8:38am UTC](https://discuss.elastic.co/t/bug-parsing-json/50921/1 "2016-05-25T08:38:59Z")
Hi everybody After search a lot about this problem I get the conclusion that this is a bug for logstash. I have asked in this forum and unfortunately I didn't got any answer, so I decided to give step by step the way to reproduce the bug. Please if this is not the right place to report a bug, tell me where. In resum I get the bug when I try to import data from a table in postgresql, that have a field type json, to an index in elasticsearch using logstash. My aim will be to get a nested field in elasticsearch index using as source the field type json in the postgres table. To reproduce the error, please following this next stepes: 1. create a table in postgresql: CREATE TABLE test\_jsonfield ( customer integer NOT NULL, categories\_json json ); 2. insert 2 records in the table INSERT INTO test\_jsonfield VALUES (1, '[{"first\_level":297,"second\_level":null}]'); INSERT INTO test\_jsonfield VALUES (2, '[{"first\_level":585,"second\_level":[1559,2445]},{"first\_level":987,"second\_level":[2]}]'); 3. Create the logstash configuration file input { jdbc { jdbc\_connection\_string =\> "jdbc:postgresql://mydomain:5432/mydatabase" jdbc\_user =\> "postgres" jdbc\_password =\> "mypassword" jdbc\_paging\_enabled =\> true jdbc\_page\_size =\> "50000" jdbc\_validate\_connection =\> true jdbc\_driver\_library =\> "/usr/share/elasticsearch/lib/postgresql-9.4.1208.jar" jdbc\_driver\_class =\> "org.postgresql.Driver" statement =\> "SELECT \* FROM test\_jsonfield" } } filter { json { source =\> "categories\_json" target =\> "categories" remove\_field =\> ["categories\_json"] } } output { elasticsearch { document\_id =\> "%{customer}" index =\> "test\_jsonfield\_nested" document\_type =\> "test" } } 1. Create the mapping for the index "test\_jsonfield\_nested" POST test\_jsonfield\_nested/ { "mappings": { "test": { "properties": { "customer": { "type": "string" }, "categories": { "type": "nested", "properties": { "first\_level": { "type": "integer" }, "second\_level": { "type": "integer" } } } } } } } 2. Check the mapping: { "test\_jsonfield\_nested": { "mappings": { "test": { "properties": { "categories": { "type": "nested", "properties": { "first\_level": { "type": "integer" }, "second\_level": { "type": "integer" } } }, "customer": { "type": "string" } } } } } } 3. run logstash sh logstash -f test\_jsonfield.conf in this point I get the following Errors: Settings: Default pipeline workers: 3 Pipeline main started Error parsing json {:source=\>"categories\_json", :raw=\>#Java::OrgPostgresqlUtil::PGobject:0x62f1e143, :exception=\>java.lang.ClassCastException: org.jruby.java.proxies.ConcreteJavaProxy cannot be cast to org.jruby.RubyIO, :level=\>:warn} Error parsing json {:source=\>"categories\_json", :raw=\>#Java::OrgPostgresqlUtil::PGobject:0x66c5829a, :exception=\>java.lang.ClassCastException: org.jruby.java.proxies.ConcreteJavaProxy cannot be cast to org.jruby.RubyIO, :level=\>:warn} Pipeline main has been shutdown stopping pipeline {:id=\>"main"} The data is imported but not as a nested field. I have expected a field name "categories" as a nested field, according to the configuration of filter "json" using in the config file "test\_jsonfield.conf" of logstash and the mapping of the index. Instead of them I get a docu with a field called "categories\_json(the same field that have in the postgres table) , something like this: "\_source": { "customer": 2, "categories\_json": { "type": "json", "value": "[{"first\_level":585,"second\_level":[1559,2445]},{"first\_level":987,"second\_level":[2]}]" }, In advance thanks for your support. Regards Jorge von Rudno ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 25, 2016, 10:37am UTC](https://discuss.elastic.co/t/bug-parsing-json/50921/2 "2016-05-25T10:37:42Z")
Oh, so the top-level content of the JSON string is an array and not an object? That could definitely be a problem. ---
**Author:** ![JvonRudno](https://avatars.discourse-cdn.com/v4/letter/j/258eb7/32.png) [@JvonRudno](https://discuss.elastic.co/u/JvonRudno)\ **Post date:** [May 25, 2016, 11:03am UTC](https://discuss.elastic.co/t/bug-parsing-json/50921/3 "2016-05-25T11:03:05Z")
Hi Magnusbaeck, Thanks for your replay!!! I have tested changing the filter to use the field ""[categories\_json][value]" as source: filter { json { source =\> "[categories\_json][value]" target =\> "categories" remove\_field =\> "categories\_json" } } And now I ger an error: Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=\>#\>, "backtrace"=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.2-java/lib/logstash/util/accessors.rb:56:in`get'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.2-java/lib/logstash/event.rb:122:in `[]'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-json-2.0.6/lib/logstash/filters/json.rb:69:in`filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:151:in `multi_filter'", "org/jruby/RubyArray.java:1613:in`each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:148:in `multi_filter'", "(eval):41:in`filter\_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:267:in `filter_batch'", "org/jruby/RubyArray.java:1613:in`each'", "org/jruby/RubyEnumerable.java:852:in `inject'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:265:in`filter\_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:223:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:201:in`start\_workers'"], :level=\>:error} NoMethodError: undefined method `[]' for #Java::OrgPostgresqlUtil::PGobject:0x37ff123 get at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.2-java/lib/logstash/util/accessors.rb:56 [] at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.3.2-java/lib/logstash/event.rb:122 filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-json-2.0.6/lib/logstash/filters/json.rb:69 multi\_filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:151 each at org/jruby/RubyArray.java:1613 multi\_filter at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/filters/base.rb:148 filter\_func at (eval):41 filter\_batch at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:267 each at org/jruby/RubyArray.java:1613 inject at org/jruby/RubyEnumerable.java:852 filter\_batch at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:265 worker\_loop at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:223 start\_workers at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:201 Regards Jorge von Rudno ---
**Author:** ![JvonRudno](https://avatars.discourse-cdn.com/v4/letter/j/258eb7/32.png) [@JvonRudno](https://discuss.elastic.co/u/JvonRudno)\ **Post date:** [June 2, 2016, 9:33am UTC](https://discuss.elastic.co/t/bug-parsing-json/50921/4 "2016-06-02T09:33:18Z")
Hi Magnusbaeck, May I ask you about the estimate time that can take to solve this issue and if perhaps you have some alternative to bypass this problems while it will solve. Regards Jorge ---
**Author:** ![purbon](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@purbon](https://discuss.elastic.co/u/purbon)\ **Post date:** [June 2, 2016, 10:17am UTC](https://discuss.elastic.co/t/bug-parsing-json/50921/5 "2016-06-02T10:17:10Z")
@JvonRudno as @magnusbaeck said you should check your json, this is not valid one. on the other side, we're here to help, so please don't ask for estimates. To move forward I would recommend you testing your json values with a pipeline that is basically stdin -\> filters -\> stdout (codec rubydebug). when this is ok, you could introduce the database back. hope this helps. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:54am UTC](https://discuss.elastic.co/t/bug-parsing-json/50921/6 "2017-07-06T04:54:50Z")