# Logstash : parse json input from http poller failing **URL:** \ **Category:** Logstash\ **Created:** [February 7, 2024, 5:06pm UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787 "2024-02-07T17:06:20Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Rasheed](https://avatars.discourse-cdn.com/v4/letter/r/ee59a6/32.png) [@Rasheed](https://discuss.elastic.co/u/Rasheed)\ **Post date:** [February 7, 2024, 5:06pm UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787/1 "2024-02-07T17:06:20Z")
I have a logstash configuration of http poller input, and elastic output, but i am struggling to store the json input from poller to index as documents. it stores the entire json as a single field but i need to store each values as a field ex input from http poller { "results": [ { "tables": [ { "rows": [ { "Tree Details[id]": "1984", "Tree Details[year]": "2018", "Tree Details[quarter\_1]": null }, { "KPI Tree Details[id]": "1984", "KPI Tree Details[year]": "2018", "KPI Tree Details[quarter\_1]": null } ] } ] } ] } without any filter it stores the results as a field with entire json in single field but i would like to parse each result of rows in each document of an index. Any suggestion is appreciated i have tried split, json\_encode and json end up with either invalid field reference or typecase error. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 7, 2024, 6:31pm UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787/2 "2024-02-07T18:31:34Z")
You could try ``` json { source => "message" remove_field => ["message"] } split { field => results } split { field => "[results][tables]" } split { field => "[results][tables][rows]" } ``` and then use ruby to move the fields in [results][tables][rows] to the top level, as shown [here](https://discuss.elastic.co/t/how-to-dynamically-move-nested-key-value-to-root-level/180006/2). ---
**Author:** ![Rasheed](https://avatars.discourse-cdn.com/v4/letter/r/ee59a6/32.png) [@Rasheed](https://discuss.elastic.co/u/Rasheed)\ **Post date:** [February 7, 2024, 6:59pm UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787/3 "2024-02-07T18:59:44Z")
thank you for your prompt reply. however looks like i ran into another problem Ruby exception occurred: Invalid FieldReference: `KPI Tree Details[gender_en]` {:class=\>"RuntimeError", :backtrace=\>["(ruby filter code):4:in `block in register'", "org/jruby/RubyHash.java:1601:in `each'", "(ruby filter code):3:in `block in register'", "C:/logstash-8.12.0/vendor/bundle/jruby/3.1.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in `inline\_script'", "C:/logstash-8.12.0/vendor/bundle/jruby/3.1.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in `filter'", "C:/logstash-8.12.0/logstash-core/lib/logstash/filters/base.rb:158:in `do\_filter'", "C:/logstash-8.12.0/logstash-core/lib/logstash/filters/base.rb:176:in `block in multi_filter'", "org/jruby/RubyArray.java:1989:in `each'", "C:/logstash-8.12.0/logstash-core/lib/logstash/filters/base.rb:173:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133:in `multi\_filter'", "C:/logstash-8.12.0/logstash-core/lib/logstash/java\_pipeline.rb:304:in `block in start\_workers'"]} i made a slight change in the filter suggested by you to apply ruby. filter { json{ source =\> "message" remove\_field =\> ["message"] } split{ field =\> results } split{ field =\> "[results][tables]" } split{ field =\> "[results][tables][rows]" target =\> doc } ruby { code =\> ' event.get("doc").each { |k, v| event.set(k,v) } event.remove("doc") ' } } Can you please help me on this ? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 7, 2024, 7:07pm UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787/4 "2024-02-07T19:07:04Z")
Remove the target option on the last split filter and use the following ruby ``` ruby { code => ' event.remove("[results][tables][rows]").each { |k, v| event.set(k,v) } ' } ``` ---
**Author:** ![Rasheed](https://avatars.discourse-cdn.com/v4/letter/r/ee59a6/32.png) [@Rasheed](https://discuss.elastic.co/u/Rasheed)\ **Post date:** [February 7, 2024, 7:21pm UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787/5 "2024-02-07T19:21:19Z")
filter { json{ source =\> "message" remove\_field =\> ["message"] } split{ field =\> results } split{ field =\> "[results][tables]" } split{ field =\> "[results][tables][rows]" } ruby { code =\> ' event.remove("[results][tables][rows]").each { |k, v| event.set(k,v) } ' } } Ruby exception occurred: Invalid FieldReference: `KPI Tree Details[quarter_en]` {:class=\>"RuntimeError" same kind of exception. Am i missing anything here ? thanks again ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 7, 2024, 8:34pm UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787/6 "2024-02-07T20:34:10Z")
> [@Rasheed](#): > > "KPI Tree Details[id]" That's going to be a problem. What do you want that field to be called? It's name cannot contain square brackets, since that is taken as a field reference. We could make it a nested field `[KPI Tree Details][id]` etc. ---
**Author:** ![Rasheed](https://avatars.discourse-cdn.com/v4/letter/r/ee59a6/32.png) [@Rasheed](https://discuss.elastic.co/u/Rasheed)\ **Post date:** [February 7, 2024, 9:10pm UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787/7 "2024-02-07T21:10:14Z")
KPI Tree Details[id] - in this if we can replace the KPI Tree Details[id] with id. that works for me My rows data is huge. 200+ fields present in each row ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 7, 2024, 10:37pm UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787/8 "2024-02-07T22:37:01Z")
Here is code to do it either way... ``` ruby { code => ' event.remove("[results][tables][rows]").each { |k, v| # Keep Tree Details as outer object newk = k.gsub(/\[/, "][").gsub(/^/, "[") # Just keep inner field names #newk = k.gsub(/.*\[/, "").gsub(/\]/, "") event.set(newk,v) } ' } ``` ---
**Author:** ![Rasheed](https://avatars.discourse-cdn.com/v4/letter/r/ee59a6/32.png) [@Rasheed](https://discuss.elastic.co/u/Rasheed)\ **Post date:** [February 8, 2024, 5:46am UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787/9 "2024-02-08T05:46:53Z")
thanks. it works ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 7, 2024, 5:47am UTC](https://discuss.elastic.co/t/logstash-parse-json-input-from-http-poller-failing/352787/10 "2024-03-07T05:47:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.