# ES indexing speed decreases substantially while doing refresh **URL:** \ **Category:** Elasticsearch\ **Created:** [March 19, 2013, 9:08am UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202 "2013-03-19T09:08:51Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![shadyabhi](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)\ **Post date:** [March 19, 2013, 9:08am UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/1 "2013-03-19T09:08:51Z")
Hi, While indexes are being refreshed (checked it via hot\_threads) , indexing speed decreases substantially. Is there a way I can schedule refreshes? Also, my ES logs are occasionally filled with this (keeps getting logged every few seconds) and indexing speed decreases substantially. [2013-03-19 09:03:28,186][INFO][cluster.metadata] [index11] [logstash-2013.03.19] update\_mapping [syslog] (dynamic) What does the above error actually mean? Is this because of too many fields coming to ES? I enabled a kv filter in logstash that created too may fields, it's related I think. -- Regards, Abhijeet Rastogi (shadyabhi) > **[Abhijeet Rastogi - Google+](https://plus.google.com/107316377741966576356/)** > > A Google+ profile for Abhijeet Rastogi -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 19, 2013, 12:52pm UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/2 "2013-03-19T12:52:29Z")
On Tue, 2013-03-19 at 14:38 +0530, Abhijeet Rastogi wrote: > Hi, > > While indexes are being refreshed (checked it via hot\_threads) , > indexing speed decreases substantially. Is there a way I can schedule > refreshes? > > Also, my ES logs are occasionally filled with this (keeps getting > logged every few seconds) and indexing speed decreases substantially. > > [2013-03-19 09:03:28,186][INFO][cluster.metadata] [index11] > [logstash-2013.03.19] update\_mapping [syslog] (dynamic) This means that you have indexed a field that hasn't been seen previously, which updates the cluster state, and can cause a pause. If you have a limited number of fields, then eventually you will have seen them all and it won't do this anymore. but if you keep adding new fields, then this will continue to happen. I think this is more likely to be causing pauses than refreshes clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shadyabhi](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)\ **Post date:** [March 20, 2013, 11:49am UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/3 "2013-03-20T11:49:35Z")
What I'm essentially doing is splitting all texts that are like key=value and creating fields key with value as value. Documents are coming in at 8000 per sec [ new index is created everyday & has size of around 500GB with standard analyzer on all fields] and so, I might be getting many many new fields. So, I guess, I should not do this until I've enough resources (Right now, only 2 nodes, 48GB RAM, 8 core 2.0GHz box). Can you give any idea about the number of instances and configuration I would need to add around 20k docs per sec with this kind of setup. I'm essentially logging logs from various machines in elasticsearch. On Tue, Mar 19, 2013 at 6:22 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > On Tue, 2013-03-19 at 14:38 +0530, Abhijeet Rastogi wrote: > > > Hi, > > > > While indexes are being refreshed (checked it via hot\_threads) , > > indexing speed decreases substantially. Is there a way I can schedule > > refreshes? > > > > Also, my ES logs are occasionally filled with this (keeps getting > > logged every few seconds) and indexing speed decreases substantially. > > > > [2013-03-19 09:03:28,186][INFO][cluster.metadata] [index11] > > [logstash-2013.03.19] update\_mapping [syslog] (dynamic) > > This means that you have indexed a field that hasn't been seen > previously, which updates the cluster state, and can cause a pause. If > you have a limited number of fields, then eventually you will have seen > them all and it won't do this anymore. but if you keep adding new > fields, then this will continue to happen. > > I think this is more likely to be causing pauses than refreshes > > clint > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Regards, Abhijeet Rastogi (shadyabhi) > **[New community features for Google Chat and an update on Currents](https://workspaceupdates.googleblog.com/2023/04/new-community-features-for-google-chat-and-an-update-currents%20.html)** > > Note:  This blog post outlines upcoming changes to Google Currents for Workspace users. For information on the previous deprecation of Googl... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 20, 2013, 12:27pm UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/4 "2013-03-20T12:27:46Z")
On Wed, 2013-03-20 at 17:19 +0530, Abhijeet Rastogi wrote: > What I'm essentially doing is splitting all texts that are like > key=value and creating fields key with value as value. > > Documents are coming in at 8000 per sec [ new index is created > everyday & has size of around 500GB with standard analyzer on all > fields] and so, I might be getting many many new fields. So, I guess, > I should not do this until I've enough resources (Right now, only 2 > nodes, 48GB RAM, 8 core 2.0GHz box). > > Can you give any idea about the number of instances and configuration > I would need to add around 20k docs per sec with this kind of setup. > I'm essentially logging logs from various machines in elasticsearch. The answer to these questions is always the same, I'm afraid: try it and see 🙂 There are just too many moving parts to give a definitive answer. A typical test is: 1. create an index with a single shard 2. pump data into it 3. while querying it 4. figure out where index/query performance drops off that pretty much gives you the limit for a single shard, which you can then extrapolate to match your future requirements clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shadyabhi](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)\ **Post date:** [March 20, 2013, 4:12pm UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/5 "2013-03-20T16:12:04Z")
Are you essentially saying that I check my limits for a single box by creating one node with single shard. Get that magical value, divide it by my required throughput & I'll get the number of nodes? On Wed, Mar 20, 2013 at 5:57 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > On Wed, 2013-03-20 at 17:19 +0530, Abhijeet Rastogi wrote: > > > What I'm essentially doing is splitting all texts that are like > > key=value and creating fields key with value as value. > > > > Documents are coming in at 8000 per sec [ new index is created > > everyday & has size of around 500GB with standard analyzer on all > > fields] and so, I might be getting many many new fields. So, I guess, > > I should not do this until I've enough resources (Right now, only 2 > > nodes, 48GB RAM, 8 core 2.0GHz box). > > > > Can you give any idea about the number of instances and configuration > > I would need to add around 20k docs per sec with this kind of setup. > > I'm essentially logging logs from various machines in elasticsearch. > > The answer to these questions is always the same, I'm afraid: try it and > see 🙂 > > There are just too many moving parts to give a definitive answer. > > A typical test is: > > 1. create an index with a single shard > 2. pump data into it > 3. while querying it > 4. figure out where index/query performance drops off > > that pretty much gives you the limit for a single shard, which you can > then extrapolate to match your future requirements > > clint > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Regards, Abhijeet Rastogi (shadyabhi) > **[New community features for Google Chat and an update on Currents](https://workspaceupdates.googleblog.com/2023/04/new-community-features-for-google-chat-and-an-update-currents%20.html)** > > Note:  This blog post outlines upcoming changes to Google Currents for Workspace users. For information on the previous deprecation of Googl... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 21, 2013, 10:26am UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/6 "2013-03-21T10:26:32Z")
On Wed, 2013-03-20 at 21:42 +0530, Abhijeet Rastogi wrote: > Are you essentially saying that I check my limits for a single box by > creating one node with single shard. Get that magical value, divide it > by my required throughput & I'll get the number of nodes? yes. more or less. add a bit of overcapacity just in case clint > On Wed, Mar 20, 2013 at 5:57 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > > > On Wed, 2013-03-20 at 17:19 +0530, Abhijeet Rastogi wrote: > > > > > What I'm essentially doing is splitting all texts that are like > > > key=value and creating fields key with value as value. > > > > > > Documents are coming in at 8000 per sec [ new index is created > > > everyday & has size of around 500GB with standard analyzer on all > > > fields] and so, I might be getting many many new fields. So, I guess, > > > I should not do this until I've enough resources (Right now, only 2 > > > nodes, 48GB RAM, 8 core 2.0GHz box). > > > > > > Can you give any idea about the number of instances and configuration > > > I would need to add around 20k docs per sec with this kind of setup. > > > I'm essentially logging logs from various machines in elasticsearch. > > > > The answer to these questions is always the same, I'm afraid: try it and > > see 🙂 > > > > There are just too many moving parts to give a definitive answer. > > > > A typical test is: > > > > 1. create an index with a single shard > > 2. pump data into it > > 3. while querying it > > 4. figure out where index/query performance drops off > > > > that pretty much gives you the limit for a single shard, which you can > > then extrapolate to match your future requirements > > > > clint > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Regards, > Abhijeet Rastogi (shadyabhi) > [Google Workspace Updates: New community features for Google Chat and an update on Currents](https://plus.google.com/107316377741966576356/) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shadyabhi](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)\ **Post date:** [March 21, 2013, 10:30am UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/7 "2013-03-21T10:30:26Z")
Is it possible in ES to reduce the effect of increase in number of indices? I noticed that as soon as the indices reaches the order of TBs, the vm heap started becoming an issue. There should be a feature like, don't keep certain indices in cache, no matter what? Or running a cron that clears cache for certain indices is the only way? On Thu, Mar 21, 2013 at 3:56 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > On Wed, 2013-03-20 at 21:42 +0530, Abhijeet Rastogi wrote: > > > Are you essentially saying that I check my limits for a single box by > > creating one node with single shard. Get that magical value, divide it > > by my required throughput & I'll get the number of nodes? > > yes. more or less. add a bit of overcapacity just in case > > clint > > > On Wed, Mar 20, 2013 at 5:57 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > > > > > On Wed, 2013-03-20 at 17:19 +0530, Abhijeet Rastogi wrote: > > > > > > > What I'm essentially doing is splitting all texts that are like > > > > key=value and creating fields key with value as value. > > > > > > > > Documents are coming in at 8000 per sec [ new index is created > > > > everyday & has size of around 500GB with standard analyzer on all > > > > fields] and so, I might be getting many many new fields. So, I guess, > > > > I should not do this until I've enough resources (Right now, only 2 > > > > nodes, 48GB RAM, 8 core 2.0GHz box). > > > > > > > > Can you give any idea about the number of instances and configuration > > > > I would need to add around 20k docs per sec with this kind of setup. > > > > I'm essentially logging logs from various machines in elasticsearch. > > > > > > The answer to these questions is always the same, I'm afraid: try it and > > > see 🙂 > > > > > > There are just too many moving parts to give a definitive answer. > > > > > > A typical test is: > > > > > > 1. create an index with a single shard > > > 2. pump data into it > > > 3. while querying it > > > 4. figure out where index/query performance drops off > > > > > > that pretty much gives you the limit for a single shard, which you can > > > then extrapolate to match your future requirements > > > > > > clint > > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > Regards, > > Abhijeet Rastogi (shadyabhi) > > [Google Workspace Updates: New community features for Google Chat and an update on Currents](https://plus.google.com/107316377741966576356/) > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Regards, Abhijeet Rastogi (shadyabhi) [http://blog.abhijeetr.com](http://blog.abhijeetr.com) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 21, 2013, 11:43am UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/8 "2013-03-21T11:43:29Z")
On Thu, 2013-03-21 at 16:00 +0530, Abhijeet Rastogi wrote: > Is it possible in ES to reduce the effect of increase in number of indices? > > I noticed that as soon as the indices reaches the order of TBs, the vm > heap started becoming an issue. There should be a feature like, don't > keep certain indices in cache, no matter what? Close unused indices? Add more RAM? Add more nodes? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shadyabhi](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)\ **Post date:** [March 21, 2013, 1:03pm UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/9 "2013-03-21T13:03:24Z")
Close unused indices is a good thing I learned. Thanks. But, how about clearning caches for old indices as a cron? I still want data to be searchable if someone searches. On Thu, Mar 21, 2013 at 5:13 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > On Thu, 2013-03-21 at 16:00 +0530, Abhijeet Rastogi wrote: > > Close unused indices? > Add more RAM? > Add more nodes? > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Regards, Abhijeet Rastogi (shadyabhi) [http://blog.abhijeetr.com](http://blog.abhijeetr.com) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 21, 2013, 3:52pm UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/10 "2013-03-21T15:52:00Z")
On Thu, 2013-03-21 at 18:33 +0530, Abhijeet Rastogi wrote: > Close unused indices is a good thing I learned. Thanks. But, how about > clearning caches for old indices as a cron? I still want data to be > searchable if someone searches. You can clear caches if you want to, but are you sure you're not optimzing prematurely? Elasticsearch should take care of this stuff itself. clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:45am UTC](https://discuss.elastic.co/t/es-indexing-speed-decreases-substantially-while-doing-refresh/11202/11 "2017-07-06T02:45:20Z")