# Having issues parsing json **URL:** \ **Category:** Logstash\ **Created:** [May 30, 2019, 10:01pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619 "2019-05-30T22:01:19Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![az123](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@az123](https://discuss.elastic.co/u/az123)\ **Post date:** [May 30, 2019, 10:01pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/1 "2019-05-30T22:01:19Z")
I am seeing \_jsonparsefailure when using json codec in logstash. ``` "message" => "%{\"node_id_str\":\"xxxxxxx\",\"subscription_id_str\":\"Sub1\",\"encoding_path\":\"Cisco-IOS-XR-fib-common-oper:fib/nodes/node/protocols/protocol/vrfs/vrf/summary\",\"collection_id\":\"111078\",\"collection_start_time\":\"1559252578037\",\"msg_timestamp\":\"1559252578821\",\"data_json\":[{\"timestamp\":\"1559252578820\",\"keys\":[{\"node-name\":\"0/6/CPU1\"},{\"protocol-name\":\"ipv4\"},{\"vrf-name\":\"default\"}],\"content\":{\"pr ``` logstash config : input { tcp { port =\> 57500 codec =\> json } } output { stdout { codec =\> rubydebug } } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 31, 2019, 12:13am UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/2 "2019-05-31T00:13:20Z")
That message is not valid JSON. The % at the start of [message] needs to be removed, and it appears to be truncated. You could use mutate+gsub to remove the %. Hard to tell what would fix the truncation, possibly a multiline codec, but that would then require at least one more gsub to remove the newlines. ---
**Author:** ![az123](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@az123](https://discuss.elastic.co/u/az123)\ **Post date:** [May 31, 2019, 12:38pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/3 "2019-05-31T12:38:21Z")
Thanks for the info. I will give that a try. ---
**Author:** ![az123](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@az123](https://discuss.elastic.co/u/az123)\ **Post date:** [May 31, 2019, 2:06pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/4 "2019-05-31T14:06:53Z")
I tried this filter: filter { mutate { gsub =\> ["fieldname", "%", " "] } } Logstash fails to parse the config and would not start! [INFO] 2019-05-31 14:03:22.115 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=\>"7.1.1"} [ERROR] 2019-05-31 14:03:23.761 [Converge PipelineAction::Create] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at line 12, column 20 (byte 252) after input {\n#via TCP encoded as JSON on port 57500 - \n tcp {\n port =\> 57500\n codec =\> json\n } \n tcp {\n port =\> 5432\n codec =\> json\n }\n filter {\n mutate ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in`initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in`block in converge\_state'"]} [INFO] 2019-05-31 14:03:24.239 [LogStash::Runner] runner - Logstash shut down. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 31, 2019, 2:37pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/5 "2019-05-31T14:37:34Z")
> [@az123](#): > > Expected one of #, =\> at line 12, column 20 (byte 252) after input {\n#via TCP encoded as JSON on port 57500 - \n tcp {\n port =\> 57500\n codec =\> json\n } \n tcp {\n port =\> 5432\n codec =\> json\n }\n filter {\n mutate " You are missing a } to close the input {} section. ---
**Author:** ![az123](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@az123](https://discuss.elastic.co/u/az123)\ **Post date:** [May 31, 2019, 2:51pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/6 "2019-05-31T14:51:27Z")
Hi, This is my config file: input { tcp { port =\> 57500 codec =\> json } filter { mutate { gsub =\> ["fieldname", "%", " "] } } } output { stdout { codec =\> rubydebug } } Error: [INFO] 2019-05-31 14:46:20.084 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=\>"7.1.1"} [ERROR] 2019-05-31 14:46:21.709 [Converge PipelineAction::Create] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at line 7, column 19 (byte 96) after input {\n tcp {\n port =\> 57500\n codec =\> json\n } \n filter {\n mutate ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in`initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in`block in converge\_state'"]} [INFO] 2019-05-31 14:46:22.201 [LogStash::Runner] runner - Logstash shut down. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 31, 2019, 3:01pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/7 "2019-05-31T15:01:12Z")
> [@az123](#): > > input { > tcp { > port =\> 57500 > codec =\> json > } > filter { As I said, you are missing a } to close the input section. Change this to ``` input { tcp { port => 57500 codec => json } } filter { ``` ---
**Author:** ![az123](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@az123](https://discuss.elastic.co/u/az123)\ **Post date:** [June 3, 2019, 1:16pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/8 "2019-06-03T13:16:07Z")
Hi, I was thinking that the filter should be a part of the input section. Anyways, moving the filter section out of the input section in the config file seem to solve this issue. Thanks for that. I still see the json parse error with the filter setup as below: ``` "tags" => [ [0] "_jsonparsefailure" ] ``` filter { mutate { gsub =\> ["fieldname", "%", ""] auto\_flush\_interval =\> 5 } } Another issue I noticed is that logstash does not seem to be doing anything until I hit ctrl + c. I tried the --pipeline.unsafe\_shutdown option but it does not help. More importantly, I noticed none of this tcp json data is showing up in the kibana so not sure if its getting into elasticsearch or not. If there is jsonparse failure in logstash will the data not get published into elasticsearch? How do I check what data is getting into elasticsearch from the cli? My output plugin configuration is below: output { elasticsearch { hosts =\> ["[http://localhost:9200](http://localhost:9200)"] index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } stdout { codec =\> rubydebug } } ---
**Author:** ![az123](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@az123](https://discuss.elastic.co/u/az123)\ **Post date:** [June 3, 2019, 1:30pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/9 "2019-06-03T13:30:27Z")
The error I see after hitting ctrl+c is below: \"throttled-packets-received\":0,\"parity-packets-received\":0,\"unknown-protocol-packets-received\":0,\"input-errors\":0,\"crc-errors\":0,\"input-overruns\":0,\"framing-errors-received\":0,\"input-ignored-packets\":0,\"input-aborts\":0,\"output-errors\":0,\"output-underruns\":0,\"output-buffer-failures\":0,\"output-buffers-swapped-out\":0,\"applique\":0,\"resets\":0,\"carrier-transitions\":0,\"availability-flag\":0,\"last-data-time\":1559568412,\"seconds-since-last-clear-counters\":0,\"last-discontinuity-time\":1554322252,\"seconds-since-packet-received\":4294967295,\"seconds-since-packet-sent\":4294967295}}],\"collection\_end\_time\":\"1559568413117\"}", "@timestamp" =\> 2019-06-03T13:26:55.294Z, "tags" =\> [ [0] "\_jsonparsefailure" ], "host" =\> “xxxxxxxxx.”, "@version" =\> "1" } [WARN] 2019-06-03 13:26:57.096 [Ruby-0-Thread-18: :1] runner - Received shutdown signal, but pipeline is still waiting for in-flight events to be processed. Sending another ^C will force quit Logstash, but this may cause data loss. [WARN] 2019-06-03 13:26:57.465 [Ruby-0-Thread-20: :1] ShutdownWatcherExt - {"inflight\_count"=\>0, "stalling\_threads\_info"=\>{"other"=\>[{"thread\_id"=\>24, "name"=\>"[main]\>worker0", "current\_call"=\>"[...]/logstash-core/lib/logstash/java\_pipeline.rb:235:in `block in start_workers'"}, {"thread_id"=>25, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:235:in`block in start\_workers'"}, {"thread\_id"=\>26, "name"=\>"[main]\>worker2", "current\_call"=\>"[...]/logstash-core/lib/logstash/java\_pipeline.rb:235:in `block in start_workers'"}, {"thread_id"=>27, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:235:in`block in start\_workers'"}, {"thread\_id"=\>28, "name"=\>"[main]\>worker4", "current\_call"=\>"[...]/logstash-core/lib/logstash/java\_pipeline.rb:235:in `block in start_workers'"}, {"thread_id"=>29, "name"=>"[main]>worker5", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:235:in`block in start\_workers'"}, {"thread\_id"=\>30, "name"=\>"[main]\>worker6", "current\_call"=\>"[...]/logstash-core/lib/logstash/java\_pipeline.rb:235:in `block in start_workers'"}, {"thread_id"=>31, "name"=>"[main]>worker7", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:235:in`block in start\_workers'"}]}} [ERROR] 2019-06-03 13:26:57.471 [Ruby-0-Thread-20: :1] ShutdownWatcherExt - The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information. [INFO] 2019-06-03 13:26:57.646 [Converge PipelineAction::Stop] javapipeline - Pipeline terminated {"pipeline.id"=\>"main"} [INFO] 2019-06-03 13:26:57.653 [LogStash::Runner] runner - Logstash shut down. ---
**Author:** ![az123](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@az123](https://discuss.elastic.co/u/az123)\ **Post date:** [June 19, 2019, 3:21pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/10 "2019-06-19T15:21:40Z")
So is there any setting that I can use in the config file to make logstash output to elasticsearch as and when new inputs are received on the pipeline? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 19, 2019, 3:44pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/11 "2019-06-19T15:44:34Z")
That is the normal mode of operation. A tcp input reads lines of text. What are you using to send lines of text to port 57500. Also, your elasticsearch index name references fields in [@metadata] -- what makes you think those fields will exist? ---
**Author:** ![az123](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@az123](https://discuss.elastic.co/u/az123)\ **Post date:** [June 19, 2019, 3:56pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/12 "2019-06-19T15:56:20Z")
Not sure I understand your question. The problem I have is tcp data does not seem to go to through the stack until ctrl+c is issued. I have syslog data (udp) going through the pipeline fine as soon as it is received. How do I get around this problem? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 19, 2019, 4:17pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/13 "2019-06-19T16:17:05Z")
> [@az123](#): > > The problem I have is tcp data does not seem to go to through the stack until ctrl+c is issued. That could be that there are no lines in the input data. What are you using to send data to the tcp input? ---
**Author:** ![az123](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@az123](https://discuss.elastic.co/u/az123)\ **Post date:** [June 19, 2019, 4:54pm UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/14 "2019-06-19T16:54:37Z")
The tcp input is a stream of json data. I am removing some spurious chars in it before making it json in the pipeline. How to find out what the pipeline is doing with the data received and why its not forwarding it to the rubydebug output or the elasticsearch output? ---
**Author:** ![az123](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@az123](https://discuss.elastic.co/u/az123)\ **Post date:** [June 21, 2019, 1:39am UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/16 "2019-06-21T01:39:22Z")
Hi, I tried some things today and observed this: If I use multiline codec in the input, I see the output on rubydebug and kibana. But if I use json or json\_lines or no codec at all in the tcp input - the pipeline is just collecting stuff but no output ☹ until I hit ctrl + c. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 19, 2019, 1:39am UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619/17 "2019-07-19T01:39:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.