# Elasticsearch Issue with custom json input data using logstash **URL:** \ **Category:** Elasticsearch\ **Created:** [February 18, 2015, 11:50pm UTC](https://discuss.elastic.co/t/elasticsearch-issue-with-custom-json-input-data-using-logstash/22253 "2015-02-18T23:50:10Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![jlam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jlam/32/878_2.png) [@jlam](https://discuss.elastic.co/u/jlam)\ **Post date:** [February 18, 2015, 11:50pm UTC](https://discuss.elastic.co/t/elasticsearch-issue-with-custom-json-input-data-using-logstash/22253/1 "2015-02-18T23:50:10Z")
Hello Everyone, I'm hoping I might get some help on how Elasticsearch. I'm getting performance issues with Elasticsearch. With our current setup: We have Elasticsearch (1.4.3), redis, and logstash installed on the same server with 30GB of memory. The ES\_HEAP\_SIZE is set to 15GB. Each server has logstash installed and push the logs to redis. The logstash on the server will pickup logs from redis and push them to Elasticsearch. We are logging apache logs on all the web servers without any performance issues. Kibana works fine and performance is pretty fast. Here is the issue: We want to do custom application logging. The logs are in json format. When Elasticsearch getting "org.elasticsearch.index.mapper.MapperParsingException: failed to parse" exceptions, the performance really degrades and become unusable. The redis will consume more and more memory. Elasticsearch will come to a point where it is doing GC. Restarting Elasticsearch doesn't help. The dataset is not that big comparing to others. The daily size of the dataset is probably 2GB to 3GB of logs. It seems that if Elasticsearch is having problem execute bulk item index, it degrades the performance considerably. I'm wondering if there are any recommendation on Elasticsearch and logstash configuration. Do I need to alter the logstash mapping? Thanks, Jared -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f9c03a63-41bc-4e73-99ec-3bf6b54cbbb5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f9c03a63-41bc-4e73-99ec-3bf6b54cbbb5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jlam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jlam/32/878_2.png) [@jlam](https://discuss.elastic.co/u/jlam)\ **Post date:** [February 18, 2015, 11:56pm UTC](https://discuss.elastic.co/t/elasticsearch-issue-with-custom-json-input-data-using-logstash/22253/2 "2015-02-18T23:56:05Z")
Also the CPU usage also jump considerably too. Thanks, Jared On Wednesday, February 18, 2015 at 3:50:11 PM UTC-8, [jl...@bills.com](mailto:jl...@bills.com) wrote: > Hello Everyone, > > I'm hoping I might get some help on how Elasticsearch. I'm getting > performance issues with Elasticsearch. > > With our current setup: > We have Elasticsearch (1.4.3), redis, and logstash installed on the same > server with 30GB of memory. The ES\_HEAP\_SIZE is set to 15GB. Each server > has logstash installed and push the logs to redis. The logstash on the > server will pickup logs from redis and push them to Elasticsearch. > > We are logging apache logs on all the web servers without any performance > issues. Kibana works fine and performance is pretty fast. > > Here is the issue: > We want to do custom application logging. The logs are in json format. > When Elasticsearch getting > "org.elasticsearch.index.mapper.MapperParsingException: failed to parse" > exceptions, the performance really degrades and become unusable. The redis > will consume more and more memory. Elasticsearch will come to a point > where it is doing GC. Restarting Elasticsearch doesn't help. > > The dataset is not that big comparing to others. The daily size of the > dataset is probably 2GB to 3GB of logs. > > It seems that if Elasticsearch is having problem execute bulk item index, > it degrades the performance considerably. > > I'm wondering if there are any recommendation on Elasticsearch and > logstash configuration. > > Do I need to alter the logstash mapping? > > Thanks, > Jared -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 19, 2015, 1:13am UTC](https://discuss.elastic.co/t/elasticsearch-issue-with-custom-json-input-data-using-logstash/22253/3 "2015-02-19T01:13:49Z")
How are you feeding the json logs into ES? On 19 February 2015 at 10:56, [jlam@bills.com](mailto:jlam@bills.com) wrote: > Also the CPU usage also jump considerably too. > > Thanks, > Jared > > On Wednesday, February 18, 2015 at 3:50:11 PM UTC-8, [jl...@bills.com](mailto:jl...@bills.com) > wrote: > > > Hello Everyone, > > > > I'm hoping I might get some help on how Elasticsearch. I'm getting > > performance issues with Elasticsearch. > > > > With our current setup: > > We have Elasticsearch (1.4.3), redis, and logstash installed on the same > > server with 30GB of memory. The ES\_HEAP\_SIZE is set to 15GB. Each server > > has logstash installed and push the logs to redis. The logstash on the > > server will pickup logs from redis and push them to Elasticsearch. > > > > We are logging apache logs on all the web servers without any performance > > issues. Kibana works fine and performance is pretty fast. > > > > Here is the issue: > > We want to do custom application logging. The logs are in json format. > > When Elasticsearch getting "org.elasticsearch.index.mapper.MapperParsingException: > > failed to parse" exceptions, the performance really degrades and become > > unusable. The redis will consume more and more memory. Elasticsearch will > > come to a point where it is doing GC. Restarting Elasticsearch doesn't > > help. > > > > The dataset is not that big comparing to others. The daily size of the > > dataset is probably 2GB to 3GB of logs. > > > > It seems that if Elasticsearch is having problem execute bulk item index, > > it degrades the performance considerably. > > > > I'm wondering if there are any recommendation on Elasticsearch and > > logstash configuration. > > > > Do I need to alter the logstash mapping? > > > > Thanks, > > Jared > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_nJqMnZSGFnw3h1LyEnxhX-17mwL7PoV1RHunhiymv5A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_nJqMnZSGFnw3h1LyEnxhX-17mwL7PoV1RHunhiymv5A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jlam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jlam/32/878_2.png) [@jlam](https://discuss.elastic.co/u/jlam)\ **Post date:** [February 19, 2015, 1:37am UTC](https://discuss.elastic.co/t/elasticsearch-issue-with-custom-json-input-data-using-logstash/22253/4 "2015-02-19T01:37:14Z")
I'm using logstash. On the client, I setup the input for logstash with json codec and output to redis server. There is a logstash instance to pop from redis list into ES. Thanks, Jared On Wednesday, February 18, 2015 at 5:14:15 PM UTC-8, Mark Walkom wrote: > How are you feeding the json logs into ES? > > On 19 February 2015 at 10:56, \<[jl...@bills.com](mailto:jl...@bills.com) \\> wrote: > > > Also the CPU usage also jump considerably too. > > > > Thanks, > > Jared > > > > On Wednesday, February 18, 2015 at 3:50:11 PM UTC-8, [jl...@bills.com](mailto:jl...@bills.com) > > wrote: > > > > > Hello Everyone, > > > > > > I'm hoping I might get some help on how Elasticsearch. I'm getting > > > performance issues with Elasticsearch. > > > > > > With our current setup: > > > We have Elasticsearch (1.4.3), redis, and logstash installed on the same > > > server with 30GB of memory. The ES\_HEAP\_SIZE is set to 15GB. Each server > > > has logstash installed and push the logs to redis. The logstash on the > > > server will pickup logs from redis and push them to Elasticsearch. > > > > > > We are logging apache logs on all the web servers without any > > > performance issues. Kibana works fine and performance is pretty fast. > > > > > > Here is the issue: > > > We want to do custom application logging. The logs are in json format. > > > When Elasticsearch getting "org.elasticsearch.index.mapper.MapperParsingException: > > > failed to parse" exceptions, the performance really degrades and become > > > unusable. The redis will consume more and more memory. Elasticsearch will > > > come to a point where it is doing GC. Restarting Elasticsearch doesn't > > > help. > > > > > > The dataset is not that big comparing to others. The daily size of the > > > dataset is probably 2GB to 3GB of logs. > > > > > > It seems that if Elasticsearch is having problem execute bulk item > > > index, it degrades the performance considerably. > > > > > > I'm wondering if there are any recommendation on Elasticsearch and > > > logstash configuration. > > > > > > Do I need to alter the logstash mapping? > > > > > > Thanks, > > > Jared > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/eaf86e7f-874c-4eef-92ae-747c305576c2%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/608d2cb9-2e54-4d47-bf92-d50ca85ca2e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/608d2cb9-2e54-4d47-bf92-d50ca85ca2e4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:31am UTC](https://discuss.elastic.co/t/elasticsearch-issue-with-custom-json-input-data-using-logstash/22253/5 "2017-07-06T00:31:43Z")