# Logstash elastic search output with very big JSON is not outputting **URL:** \ **Category:** Logstash\ **Created:** [September 25, 2017, 5:05am UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632 "2017-09-25T05:05:52Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![elasticuser1](https://avatars.discourse-cdn.com/v4/letter/e/a3d4f5/32.png) [@elasticuser1](https://discuss.elastic.co/u/elasticuser1)\ **Post date:** [September 25, 2017, 5:05am UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/1 "2017-09-25T05:05:52Z")
Hi, I have very big nested json (22KB) which I am trying to output to elastic search but it's not outputting. Output to file works as expected. I tried with small json which works. Is there any limitation on json filter? Please suggest what should I do here! Thanks in advance. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 25, 2017, 1:18pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/2 "2017-09-25T13:18:59Z")
Have you checked your Logstash logs for errors or other clues? ---
**Author:** ![elasticuser1](https://avatars.discourse-cdn.com/v4/letter/e/a3d4f5/32.png) [@elasticuser1](https://discuss.elastic.co/u/elasticuser1)\ **Post date:** [September 25, 2017, 1:25pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/3 "2017-09-25T13:25:48Z")
Hi @magnusbaeck, I don't see any errors in the logs.. Any guess? I am on 5.5.0... ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 25, 2017, 1:49pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/4 "2017-09-25T13:49:46Z")
> ``` > codec => "json" > > ``` Remove this. ---
**Author:** ![elasticuser1](https://avatars.discourse-cdn.com/v4/letter/e/a3d4f5/32.png) [@elasticuser1](https://discuss.elastic.co/u/elasticuser1)\ **Post date:** [September 25, 2017, 4:56pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/5 "2017-09-25T16:56:24Z")
Hi @magnusbaeck, I removed json codec but its still not outputting to elastic search... And logstash is not flushing the events until I stop.. Here is the log: [2017-09-25T13:04:25,175][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} ^C[2017-09-25T13:04:43,107][WARN][logstash.runner] SIGINT received. Shutting down the agent. [2017-09-25T13:04:43,124][WARN][logstash.agent] stopping pipeline {:id=\>"main"} [2017-09-25T13:04:48,114][WARN][logstash.runner] Received shutdown signal, but pipeline is still waiting for in-flight events to be processed. Sending another ^C will force quit Logstash, but this may cause data loss. [2017-09-25T13:04:48,134][WARN][logstash.shutdownwatcher] {"inflight\_count"=\>1, "stalling\_thread\_info"=\>{["LogStash::Filters::Mutate", {"remove\_field"=\>["prefix", "filename", "message"], "id"=\>"98f03816aae7e170d420da8b2950e9f999380aa9-6"}]=\>[{"thread\_id"=\>43, "name"=\>"[main]\>worker10", "current\_call"=\>"[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/response.rb:50:in `call'"}]}} [2017-09-25T13:04:48,136][ERROR][logstash.shutdownwatcher] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 25, 2017, 7:27pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/6 "2017-09-25T19:27:43Z")
Are you getting anything to your file or stdout outputs? Have you verified that your multiline configuration works? What does the input look like? ---
**Author:** ![elasticuser1](https://avatars.discourse-cdn.com/v4/letter/e/a3d4f5/32.png) [@elasticuser1](https://discuss.elastic.co/u/elasticuser1)\ **Post date:** [September 25, 2017, 8:46pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/7 "2017-09-25T20:46:35Z")
Thanks @magnusbaeck. My input looks like this (seperated by \n) - ---
**Author:** ![elasticuser1](https://avatars.discourse-cdn.com/v4/letter/e/a3d4f5/32.png) [@elasticuser1](https://discuss.elastic.co/u/elasticuser1)\ **Post date:** [September 25, 2017, 9:00pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/8 "2017-09-25T21:00:17Z")
One issue I figured out. auto\_flush\_interval is missing. Not sure what is the issue with other config... ---
**Author:** ![elasticuser1](https://avatars.discourse-cdn.com/v4/letter/e/a3d4f5/32.png) [@elasticuser1](https://discuss.elastic.co/u/elasticuser1)\ **Post date:** [September 25, 2017, 9:31pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/9 "2017-09-25T21:31:48Z")
How to specify the multi line pattern when \n is at the end of line? I configured like below and I see it's looking at starting of the line so my output is missing last flower bracket... pattern =\> "^\n" ---
**Author:** ![elasticuser1](https://avatars.discourse-cdn.com/v4/letter/e/a3d4f5/32.png) [@elasticuser1](https://discuss.elastic.co/u/elasticuser1)\ **Post date:** [September 25, 2017, 11:10pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/10 "2017-09-25T23:10:01Z")
I am still having issues with 43KB json not outputting to elastic search though its valid.. What should I do to index this big document? Logstash did not complain and I don't see errors in elastic search logs also... ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 26, 2017, 5:10am UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/11 "2017-09-26T05:10:03Z")
So your input files contain multiple multiline JSON structures, separated by blank lines? First of all I don't think `\n` in the pattern makes any sense. If you want to match a blank line use `^$`. Secondly I suspect using `what => "previous"` might be a better idea. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 26, 2017, 5:12am UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/12 "2017-09-26T05:12:19Z")
Even though the last document may be valid JSON, it can not be indexed into Elasticsearch as it contains fields with conflicting mappings. You should be seeing an error in Logstash about this. If you can not find this , you can however also try indexing a single document using curl, and this should give you an error about mapping conflict. I have not gone through the full document, but can clearly see that at least the `type` field towards the end is mapped both as a string and as an object, which is not allowed in Elasticsearch. Every field must have a single mapping thoughout the entire index. ---
**Author:** ![elasticuser1](https://avatars.discourse-cdn.com/v4/letter/e/a3d4f5/32.png) [@elasticuser1](https://discuss.elastic.co/u/elasticuser1)\ **Post date:** [September 26, 2017, 5:39pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/13 "2017-09-26T17:39:56Z")
Thanks @Christian_Dahlqvist Because it is nested json, are you saying that all properties under the same parent node should be of similar type? Can you please point me to documentation? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 24, 2017, 5:40pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-output-with-very-big-json-is-not-outputting/101632/14 "2017-10-24T17:40:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.