# CPU spike on indexing data into ElastiSearch **URL:** \ **Category:** Elasticsearch\ **Created:** [February 10, 2015, 8:23pm UTC](https://discuss.elastic.co/t/cpu-spike-on-indexing-data-into-elastisearch/22072 "2015-02-10T20:23:08Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Sagar\_Shah](https://avatars.discourse-cdn.com/v4/letter/s/a9a28c/32.png) [@Sagar\_Shah](https://discuss.elastic.co/u/Sagar_Shah)\ **Post date:** [February 10, 2015, 8:23pm UTC](https://discuss.elastic.co/t/cpu-spike-on-indexing-data-into-elastisearch/22072/1 "2015-02-10T20:23:08Z")
Hello everyone, I am new to Elastic Search world and I am evaluating it for storing and retrieval of my application logs, which is currently being stored in postgres database. For performance and disk space reasons, I plan to switch logs storage from posgres to elastic search. I plan to store log information for various applications as different indices in ES. But when I implemented this tool, I noticed lots of cpu spike on my system when it received the request for inserting logs data into ES. I am using Linux and allocated ES a 2GB RAM. I am using REST service calls to push logs data into ES. There are about 5000 records being inserted into one index for every request into the application (all in JSON format). Refresh interval is kept to 30 seconds in elasticsearch.yml file Number of shards configured to 1 and replicas to 0. There are about 20 index created for different applications, which might eventually grow upto 50 (as it is per application) Also, advise, if it should be separate index per application, or single index storing logs of all of the applications. I expect about 3k to 5k log messages per any request in any application Single node is being used for storing logs of multiple applications running on same node under separate JVM JDK: 1.7 ES\_HEAP\_SIZE=2g Also configured, mlockall to true in elasticsearch.yml file. I am clueless in identifying root cause of cpu spike. Can anyone please suggest a way out in further troubleshooting and determining the root cause. What should be the next steps to troubleshoot the same. ES version: 1.4.2 Please let me know, if you need any further information. Appreciate your inputs and help! Regards, Sagar Shah -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 10, 2015, 9:02pm UTC](https://discuss.elastic.co/t/cpu-spike-on-indexing-data-into-elastisearch/22072/2 "2015-02-10T21:02:12Z")
That would be the indexing process that happens when you send the logs, as ES needs to process and turn these into an inverted index for searching. On 11 February 2015 at 07:23, Sagar Shah [sagarshah1983@gmail.com](mailto:sagarshah1983@gmail.com) wrote: > Hello everyone, > I am new to Elastic Search world and I am evaluating it for storing and > retrieval of my application logs, which is currently being stored in > postgres database. > For performance and disk space reasons, I plan to switch logs storage from > posgres to Elasticsearch. I plan to store log information for various > applications as different indices in ES. > > But when I implemented this tool, I noticed lots of cpu spike on my system > when it received the request for inserting logs data into ES. > I am using Linux and allocated ES a 2GB RAM. I am using REST service calls > to push logs data into ES. There are about 5000 records being inserted into > one index for every request into the application (all in JSON format). > Refresh interval is kept to 30 seconds in elasticsearch.yml file > > Number of shards configured to 1 and replicas to 0. There are about 20 > index created for different applications, which might eventually grow upto > 50 (as it is per application) Also, advise, if it should be separate index > per application, or single index storing logs of all of the applications. I > expect about 3k to 5k log messages per any request in any application > Single node is being used for storing logs of multiple applications > running on same node under separate JVM > JDK: 1.7 > > ES\_HEAP\_SIZE=2g > Also configured, mlockall to true in elasticsearch.yml file. > > I am clueless in identifying root cause of cpu spike. > > Can anyone please suggest a way out in further troubleshooting and > determining the root cause. What should be the next steps to troubleshoot > the same. > > ES version: 1.4.2 > > Please let me know, if you need any further information. > Appreciate your inputs and help! > > Regards, > Sagar Shah > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9%3DVrKWjfqFLV9Wb1T7%2BZ\_iA91bqhLDegAhEdnzx5MePQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9%3DVrKWjfqFLV9Wb1T7%2BZ_iA91bqhLDegAhEdnzx5MePQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Sagar\_Shah](https://avatars.discourse-cdn.com/v4/letter/s/a9a28c/32.png) [@Sagar\_Shah](https://discuss.elastic.co/u/Sagar_Shah)\ **Post date:** [February 10, 2015, 9:23pm UTC](https://discuss.elastic.co/t/cpu-spike-on-indexing-data-into-elastisearch/22072/3 "2015-02-10T21:23:14Z")
Thanks for your input Mark. Can that be optimized? Does that run real time? I configured index.refresh\_interval to run every 30 seconds. Can I optimize indexing process anyway. I have observed that whenever I process 3-4 events in my application causing about 5-6K records insertion in elasticseach, there's cpu spike. I monitored heap as well, and its way below (200mb) size configured (2gb) in ES. I don't need to use any fancy feature of cluster or availability at this point. My whole objective (from previous post) is to ensure that I store logs information and retrieve it using API and show in my web administration application without causing any performance or cpu spike. Please advise. Thanks! Regards, Sagar Shah On Wednesday, February 11, 2015 at 2:32:45 AM UTC+5:30, Mark Walkom wrote: > That would be the indexing process that happens when you send the logs, as > ES needs to process and turn these into an inverted index for searching. > > On 11 February 2015 at 07:23, Sagar Shah \<[sagars...@gmail.com](mailto:sagars...@gmail.com) > \\> wrote: > > > Hello everyone, > > I am new to Elastic Search world and I am evaluating it for storing and > > retrieval of my application logs, which is currently being stored in > > postgres database. > > For performance and disk space reasons, I plan to switch logs storage > > from posgres to Elasticsearch. I plan to store log information for various > > applications as different indices in ES. > > > > But when I implemented this tool, I noticed lots of cpu spike on my > > system when it received the request for inserting logs data into ES. > > I am using Linux and allocated ES a 2GB RAM. I am using REST service > > calls to push logs data into ES. There are about 5000 records being > > inserted into one index for every request into the application (all in JSON > > format). Refresh interval is kept to 30 seconds in elasticsearch.yml file > > > > Number of shards configured to 1 and replicas to 0. There are about 20 > > index created for different applications, which might eventually grow upto > > 50 (as it is per application) Also, advise, if it should be separate index > > per application, or single index storing logs of all of the applications. I > > expect about 3k to 5k log messages per any request in any application > > Single node is being used for storing logs of multiple applications > > running on same node under separate JVM > > JDK: 1.7 > > > > ES\_HEAP\_SIZE=2g > > Also configured, mlockall to true in elasticsearch.yml file. > > > > I am clueless in identifying root cause of cpu spike. > > > > Can anyone please suggest a way out in further troubleshooting and > > determining the root cause. What should be the next steps to troubleshoot > > the same. > > > > ES version: 1.4.2 > > > > Please let me know, if you need any further information. > > Appreciate your inputs and help! > > > > Regards, > > Sagar Shah > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 10, 2015, 10:07pm UTC](https://discuss.elastic.co/t/cpu-spike-on-indexing-data-into-elastisearch/22072/4 "2015-02-10T22:07:38Z")
You will never get around the spike on some level. Elasticsearch uses resources at index time to ensure the searching is fast. You may be able to reduce it a little by playing with your bulk size, ie reducing it, or adding more CPUs On 11 February 2015 at 08:23, Sagar Shah [sagarshah1983@gmail.com](mailto:sagarshah1983@gmail.com) wrote: > Thanks for your input Mark. > > Can that be optimized? Does that run real time? I configured > index.refresh\_interval to run every 30 seconds. Can I optimize indexing > process anyway. > I have observed that whenever I process 3-4 events in my application > causing about 5-6K records insertion in elasticseach, there's cpu spike. > > I monitored heap as well, and its way below (200mb) size configured (2gb) > in ES. > > I don't need to use any fancy feature of cluster or availability at this > point. > My whole objective (from previous post) is to ensure that I store logs > information and retrieve it using API and show in my web administration > application without causing any performance or cpu spike. > > Please advise. > Thanks! > > Regards, > Sagar Shah > On Wednesday, February 11, 2015 at 2:32:45 AM UTC+5:30, Mark Walkom wrote: > > > That would be the indexing process that happens when you send the logs, > > as ES needs to process and turn these into an inverted index for searching. > > > > On 11 February 2015 at 07:23, Sagar Shah [sagars...@gmail.com](mailto:sagars...@gmail.com) wrote: > > > > > Hello everyone, > > > I am new to Elastic Search world and I am evaluating it for storing and > > > retrieval of my application logs, which is currently being stored in > > > postgres database. > > > For performance and disk space reasons, I plan to switch logs storage > > > from posgres to Elasticsearch. I plan to store log information for various > > > applications as different indices in ES. > > > > > > But when I implemented this tool, I noticed lots of cpu spike on my > > > system when it received the request for inserting logs data into ES. > > > I am using Linux and allocated ES a 2GB RAM. I am using REST service > > > calls to push logs data into ES. There are about 5000 records being > > > inserted into one index for every request into the application (all in JSON > > > format). Refresh interval is kept to 30 seconds in elasticsearch.yml file > > > > > > Number of shards configured to 1 and replicas to 0. There are about 20 > > > index created for different applications, which might eventually grow upto > > > 50 (as it is per application) Also, advise, if it should be separate index > > > per application, or single index storing logs of all of the applications. I > > > expect about 3k to 5k log messages per any request in any application > > > Single node is being used for storing logs of multiple applications > > > running on same node under separate JVM > > > JDK: 1.7 > > > > > > ES\_HEAP\_SIZE=2g > > > Also configured, mlockall to true in elasticsearch.yml file. > > > > > > I am clueless in identifying root cause of cpu spike. > > > > > > Can anyone please suggest a way out in further troubleshooting and > > > determining the root cause. What should be the next steps to troubleshoot > > > the same. > > > > > > ES version: 1.4.2 > > > > > > Please let me know, if you need any further information. > > > Appreciate your inputs and help! > > > > > > Regards, > > > Sagar Shah > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV\_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Sagar\_Shah](https://avatars.discourse-cdn.com/v4/letter/s/a9a28c/32.png) [@Sagar\_Shah](https://discuss.elastic.co/u/Sagar_Shah)\ **Post date:** [February 11, 2015, 6:22am UTC](https://discuss.elastic.co/t/cpu-spike-on-indexing-data-into-elastisearch/22072/5 "2015-02-11T06:22:36Z")
Thanks Mark for sharing your inputs, but I see CPU spiking upto 80% at times. Is it expected or desirable for few thousands records? On Wed, Feb 11, 2015 at 3:37 AM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > You will never get around the spike on some level. Elasticsearch uses > resources at index time to ensure the searching is fast. > > You may be able to reduce it a little by playing with your bulk size, ie > reducing it, or adding more CPUs > > On 11 February 2015 at 08:23, Sagar Shah [sagarshah1983@gmail.com](mailto:sagarshah1983@gmail.com) wrote: > > > Thanks for your input Mark. > > > > Can that be optimized? Does that run real time? I configured > > index.refresh\_interval to run every 30 seconds. Can I optimize indexing > > process anyway. > > I have observed that whenever I process 3-4 events in my application > > causing about 5-6K records insertion in elasticseach, there's cpu spike. > > > > I monitored heap as well, and its way below (200mb) size configured (2gb) > > in ES. > > > > I don't need to use any fancy feature of cluster or availability at this > > point. > > My whole objective (from previous post) is to ensure that I store logs > > information and retrieve it using API and show in my web administration > > application without causing any performance or cpu spike. > > > > Please advise. > > Thanks! > > > > Regards, > > Sagar Shah > > On Wednesday, February 11, 2015 at 2:32:45 AM UTC+5:30, Mark Walkom wrote: > > > > > That would be the indexing process that happens when you send the logs, > > > as ES needs to process and turn these into an inverted index for searching. > > > > > > On 11 February 2015 at 07:23, Sagar Shah [sagars...@gmail.com](mailto:sagars...@gmail.com) wrote: > > > > > > > Hello everyone, > > > > I am new to Elastic Search world and I am evaluating it for storing and > > > > retrieval of my application logs, which is currently being stored in > > > > postgres database. > > > > For performance and disk space reasons, I plan to switch logs storage > > > > from posgres to Elasticsearch. I plan to store log information for various > > > > applications as different indices in ES. > > > > > > > > But when I implemented this tool, I noticed lots of cpu spike on my > > > > system when it received the request for inserting logs data into ES. > > > > I am using Linux and allocated ES a 2GB RAM. I am using REST service > > > > calls to push logs data into ES. There are about 5000 records being > > > > inserted into one index for every request into the application (all in JSON > > > > format). Refresh interval is kept to 30 seconds in elasticsearch.yml file > > > > > > > > Number of shards configured to 1 and replicas to 0. There are about 20 > > > > index created for different applications, which might eventually grow upto > > > > 50 (as it is per application) Also, advise, if it should be separate index > > > > per application, or single index storing logs of all of the applications. I > > > > expect about 3k to 5k log messages per any request in any application > > > > Single node is being used for storing logs of multiple applications > > > > running on same node under separate JVM > > > > JDK: 1.7 > > > > > > > > ES\_HEAP\_SIZE=2g > > > > Also configured, mlockall to true in elasticsearch.yml file. > > > > > > > > I am clueless in identifying root cause of cpu spike. > > > > > > > > Can anyone please suggest a way out in further troubleshooting and > > > > determining the root cause. What should be the next steps to troubleshoot > > > > the same. > > > > > > > > ES version: 1.4.2 > > > > > > > > Please let me know, if you need any further information. > > > > Appreciate your inputs and help! > > > > > > > > Regards, > > > > Sagar Shah > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/bcmrhRmh\_r4/unsubscribe](https://groups.google.com/d/topic/elasticsearch/bcmrhRmh_r4/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV\_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV\_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Regards, Sagar Shah Too many people think more of security instead of opportunity. They seem more afraid of life than death!!! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANZnv5iCnbF7yhYTA7mWB1sB2%2Bt45RvGkrmf91tqjkRArLhncg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANZnv5iCnbF7yhYTA7mWB1sB2%2Bt45RvGkrmf91tqjkRArLhncg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 11, 2015, 7:24am UTC](https://discuss.elastic.co/t/cpu-spike-on-indexing-data-into-elastisearch/22072/6 "2015-02-11T07:24:48Z")
You _cannot_ get around this, it is expected, it is desirable. On 11 February 2015 at 17:22, Sagar Shah [sagarshah1983@gmail.com](mailto:sagarshah1983@gmail.com) wrote: > Thanks Mark for sharing your inputs, but I see CPU spiking upto 80% at > times. > Is it expected or desirable for few thousands records? > > On Wed, Feb 11, 2015 at 3:37 AM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > > > You will never get around the spike on some level. Elasticsearch uses > > resources at index time to ensure the searching is fast. > > > > You may be able to reduce it a little by playing with your bulk size, ie > > reducing it, or adding more CPUs > > > > On 11 February 2015 at 08:23, Sagar Shah [sagarshah1983@gmail.com](mailto:sagarshah1983@gmail.com) wrote: > > > > > Thanks for your input Mark. > > > > > > Can that be optimized? Does that run real time? I configured > > > index.refresh\_interval to run every 30 seconds. Can I optimize indexing > > > process anyway. > > > I have observed that whenever I process 3-4 events in my application > > > causing about 5-6K records insertion in elasticseach, there's cpu spike. > > > > > > I monitored heap as well, and its way below (200mb) size configured > > > (2gb) in ES. > > > > > > I don't need to use any fancy feature of cluster or availability at this > > > point. > > > My whole objective (from previous post) is to ensure that I store logs > > > information and retrieve it using API and show in my web administration > > > application without causing any performance or cpu spike. > > > > > > Please advise. > > > Thanks! > > > > > > Regards, > > > Sagar Shah > > > On Wednesday, February 11, 2015 at 2:32:45 AM UTC+5:30, Mark Walkom > > > wrote: > > > > > > > That would be the indexing process that happens when you send the logs, > > > > as ES needs to process and turn these into an inverted index for searching. > > > > > > > > On 11 February 2015 at 07:23, Sagar Shah [sagars...@gmail.com](mailto:sagars...@gmail.com) wrote: > > > > > > > > > Hello everyone, > > > > > I am new to Elastic Search world and I am evaluating it for storing > > > > > and retrieval of my application logs, which is currently being stored in > > > > > postgres database. > > > > > For performance and disk space reasons, I plan to switch logs storage > > > > > from posgres to Elasticsearch. I plan to store log information for various > > > > > applications as different indices in ES. > > > > > > > > > > But when I implemented this tool, I noticed lots of cpu spike on my > > > > > system when it received the request for inserting logs data into ES. > > > > > I am using Linux and allocated ES a 2GB RAM. I am using REST service > > > > > calls to push logs data into ES. There are about 5000 records being > > > > > inserted into one index for every request into the application (all in JSON > > > > > format). Refresh interval is kept to 30 seconds in elasticsearch.yml file > > > > > > > > > > Number of shards configured to 1 and replicas to 0. There are about 20 > > > > > index created for different applications, which might eventually grow upto > > > > > 50 (as it is per application) Also, advise, if it should be separate index > > > > > per application, or single index storing logs of all of the applications. I > > > > > expect about 3k to 5k log messages per any request in any application > > > > > Single node is being used for storing logs of multiple applications > > > > > running on same node under separate JVM > > > > > JDK: 1.7 > > > > > > > > > > ES\_HEAP\_SIZE=2g > > > > > Also configured, mlockall to true in elasticsearch.yml file. > > > > > > > > > > I am clueless in identifying root cause of cpu spike. > > > > > > > > > > Can anyone please suggest a way out in further troubleshooting and > > > > > determining the root cause. What should be the next steps to troubleshoot > > > > > the same. > > > > > > > > > > ES version: 1.4.2 > > > > > > > > > > Please let me know, if you need any further information. > > > > > Appreciate your inputs and help! > > > > > > > > > > Regards, > > > > > Sagar Shah > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa4b625b-4e5d-4f6a-b92f-9b9c23cb3e4f%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9b677166-bd29-4ef9-864a-480b5b09d565%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/bcmrhRmh\_r4/unsubscribe](https://groups.google.com/d/topic/elasticsearch/bcmrhRmh_r4/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV\_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV\_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X94dLvjrV_2Vina9Z1uas7mmxWBeDdo32zHxet9qtDq9g%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > > Regards, > Sagar Shah > > Too many people think more of security instead of opportunity. They seem > more afraid of life than death!!! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CANZnv5iCnbF7yhYTA7mWB1sB2%2Bt45RvGkrmf91tqjkRArLhncg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANZnv5iCnbF7yhYTA7mWB1sB2%2Bt45RvGkrmf91tqjkRArLhncg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CANZnv5iCnbF7yhYTA7mWB1sB2%2Bt45RvGkrmf91tqjkRArLhncg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CANZnv5iCnbF7yhYTA7mWB1sB2%2Bt45RvGkrmf91tqjkRArLhncg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-KLe5Mh62nKkYNce9ZH%3DD0qtWyfHgzUSrLJiY9PPWACA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-KLe5Mh62nKkYNce9ZH%3DD0qtWyfHgzUSrLJiY9PPWACA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:33am UTC](https://discuss.elastic.co/t/cpu-spike-on-indexing-data-into-elastisearch/22072/7 "2017-07-06T00:33:40Z")