# Extremly slow troughput on large index **URL:** \ **Category:** Elasticsearch\ **Created:** [April 9, 2014, 1:11pm UTC](https://discuss.elastic.co/t/extremly-slow-troughput-on-large-index/16897 "2014-04-09T13:11:21Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Yitzhak\_Kesselman](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yitzhak\_Kesselman](https://discuss.elastic.co/u/Yitzhak_Kesselman)\ **Post date:** [April 9, 2014, 1:11pm UTC](https://discuss.elastic.co/t/extremly-slow-troughput-on-large-index/16897/1 "2014-04-09T13:11:21Z")
Hi, I have configured a single node ES with logstash 1.4.0 (8GB memory) with the following configuration: - ## index.number\_of\_shards: 7 ## number\_of\_replicas: 0 ## refresh\_interval: -1 ## translog.flush\_threshold\_ops: 100000 ## merge.policy.merge\_factor: 30 ## codec.bloom.load: false ## min\_shard\_index\_buffer\_size: 12m ## compound\_format : true ## indices.fielddata.cache.size: 15% ## indices.fielddata.cache.expire: 5m ## indices.cache.filter.size: 15% indices.cache.filter.expire: 5m Machine : 16GB RAM, Intel I&-2600 CPU @ 3.4GHZ. OS: 64bit WindowsServer 2012 R2 My raw data is CSV file and I use grok as a filter to parse it with output configuration (elasticsearch { embedded =\> true flush\_size =\> 100000 idle\_flush\_time =\> 30 }). Row data size is about 100GB events per day which ES tries to input into one index (with 7 shards). At the beginning the insert was fast however after a while it's got extremely slow, 1.5K doc in 8K seconds ☹ Currently the index has around 140Million docs with size of 55GB. When I have analyzed the write to the disk with ProcMon I have seen that the process is writing in an interleaved manner to three kinds of files (.tim, .doc, and .pos) in 4K and 8K segments, instead of batching writes to some reasonable number. Appreciate the help. All the best, Yitzhak -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [April 9, 2014, 2:28pm UTC](https://discuss.elastic.co/t/extremly-slow-troughput-on-large-index/16897/2 "2014-04-09T14:28:11Z")
Shooting in the dark here, but here it goes: 1. Do you have anything else running on the system? for example AVs are known to cause slow-downs for such services, and other I/O or memory heavy services could cause thrashing or just general slowdown 2. What JVM version are you running this with? 3. If you changed any of the default settings for merge factors etc - can you revert that and try again? 4. Can you try with embedded=false and see if it makes a difference? -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Wed, Apr 9, 2014 at 4:11 PM, Yitzhak Kesselman [ikesselman@gmail.com](mailto:ikesselman@gmail.com)wrote: > Hi, > > I have configured a single node ES with logstash 1.4.0 (8GB memory) with > the following configuration: > > - > > ## index.number\_of\_shards: 7 > > ## number\_of\_replicas: 0 > > ## refresh\_interval: -1 > > ## translog.flush\_threshold\_ops: 100000 > > ## merge.policy.merge\_factor: 30 > > ## codec.bloom.load: false > > ## min\_shard\_index\_buffer\_size: 12m > > ## compound\_format : true > > ## indices.fielddata.cache.size: 15% > > ## indices.fielddata.cache.expire: 5m > > ## indices.cache.filter.size: 15% > > indices.cache.filter.expire: 5m > > Machine : 16GB RAM, Intel I&-2600 CPU @ 3.4GHZ. > OS: 64bit WindowsServer 2012 R2 > > My raw data is CSV file and I use grok as a filter to parse it with output > configuration (elasticsearch { embedded =\> true flush\_size =\> > 100000 idle\_flush\_time =\> 30 }). > Row data size is about 100GB events per day which ES tries to input into > one index (with 7 shards). > > At the beginning the insert was fast however after a while > it's got extremely slow, 1.5K doc in 8K seconds ☹ > > Currently the index has around 140Million docs with size of 55GB. > > When I have analyzed the write to the disk with ProcMon I have seen that > the process is writing in an interleaved manner to three kinds of files > (.tim, .doc, and .pos) in 4K and 8K segments, instead of batching writes to > some reasonable number. > > Appreciate the help. > > All the best, > > Yitzhak > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvSC%2BSJvPNoZwXbCLWSBGe1n8sJprrqjFgG6ubBoqWTZA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvSC%2BSJvPNoZwXbCLWSBGe1n8sJprrqjFgG6ubBoqWTZA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yitzhak\_Kesselman](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yitzhak\_Kesselman](https://discuss.elastic.co/u/Yitzhak_Kesselman)\ **Post date:** [April 9, 2014, 6:36pm UTC](https://discuss.elastic.co/t/extremly-slow-troughput-on-large-index/16897/3 "2014-04-09T18:36:12Z")
Answers inline. Regarding the slow I/O. When I analyzed the creation of the Lucene index files I see that they are created without any special flags (such as no buffering or write through). This means that we’re paying costs twice – when we write the file we’re going cache data in Windows’ Cache Manager, which takes a lot of memory (which is then not available to the application itself) but when we read the file we don’t actually read it using the cache, which makes the operation slow. _Any ideas?_ On Wednesday, April 9, 2014 5:28:11 PM UTC+3, Itamar Syn-Hershko wrote: > Shooting in the dark here, but here it goes: > > 1. Do you have anything else running on the system? for example AVs are > known to cause slow-downs for such services, and other I/O or memory heavy > services could cause thrashing or just general slowdown No, nothing else is running on that machine. Initially it was working fast it got slower with that amount of data that in index. Moreover is there a way to increase buffer size for the Lucene index files (.tim, .doc, and .pos) from 8K to something much bigger. > 1. What JVM version are you running this with? java version "1.7.0\_51" Java(TM) SE Runtime Environment (build 1.7.0\_51-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) OS\_NAME="Windows" OS\_VERSION="5.2" OS\_ARCH="amd64" 1. If you changed any of the default settings for merge factors etc - can > you revert that and try again? Tried before was same behavior. > 1. Can you try with embedded=false and see if it makes a difference? Tried before was same behavior. > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > On Wed, Apr 9, 2014 at 4:11 PM, Yitzhak Kesselman \<[ikess...@gmail.com](mailto:ikess...@gmail.com)\ > > > wrote: > > > Hi, > > > > I have configured a single node ES with logstash 1.4.0 (8GB memory) with > > the following configuration: > > > > - > > > > ## index.number\_of\_shards: 7 > > > > ## number\_of\_replicas: 0 > > > > ## refresh\_interval: -1 > > > > ## translog.flush\_threshold\_ops: 100000 > > > > ## merge.policy.merge\_factor: 30 > > > > ## codec.bloom.load: false > > > > ## min\_shard\_index\_buffer\_size: 12m > > > > ## compound\_format : true > > > > ## indices.fielddata.cache.size: 15% > > > > ## indices.fielddata.cache.expire: 5m > > > > ## indices.cache.filter.size: 15% > > > > indices.cache.filter.expire: 5m > > > > Machine : 16GB RAM, Intel I&-2600 CPU @ 3.4GHZ. > > OS: 64bit WindowsServer 2012 R2 > > > > My raw data is CSV file and I use grok as a filter to parse it with > > output configuration (elasticsearch { embedded =\> true flush\_size =\> > > 100000 idle\_flush\_time =\> 30 }). > > Row data size is about 100GB events per day which ES tries to input into > > one index (with 7 shards). > > > > At the beginning the insert was fast however after a while > > it's got extremely slow, 1.5K doc in 8K seconds ☹ > > > > Currently the index has around 140Million docs with size of 55GB. > > > > When I have analyzed the write to the disk with ProcMon I have seen that > > the process is writing in an interleaved manner to three kinds of files > > (.tim, .doc, and .pos) in 4K and 8K segments, instead of batching writes to > > some reasonable number. > > > > Appreciate the help. > > > > All the best, > > > > Yitzhak > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d6c4207d-8858-4440-81d6-61298fa5b0e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d6c4207d-8858-4440-81d6-61298fa5b0e4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yitzhak\_Kesselman](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yitzhak\_Kesselman](https://discuss.elastic.co/u/Yitzhak_Kesselman)\ **Post date:** [April 9, 2014, 7:29pm UTC](https://discuss.elastic.co/t/extremly-slow-troughput-on-large-index/16897/4 "2014-04-09T19:29:04Z")
I have attached the indexing rate from bigdesk: On Wednesday, April 9, 2014 9:36:12 PM UTC+3, Yitzhak Kesselman wrote: > Answers inline. > > Regarding the slow I/O. When I analyzed the creation of the Lucene index > files I see that they are created without any special flags (such as no > buffering or write through). This means that we’re paying costs twice – > when we write the file we’re going cache data in Windows’ Cache Manager, > which takes a lot of memory (which is then not available to the application > itself) but when we read the file we don’t actually read it using the > cache, which makes the operation slow. _Any ideas?_ > > On Wednesday, April 9, 2014 5:28:11 PM UTC+3, Itamar Syn-Hershko wrote: > > > Shooting in the dark here, but here it goes: > > > > 1. Do you have anything else running on the system? for example AVs are > > known to cause slow-downs for such services, and other I/O or memory heavy > > services could cause thrashing or just general slowdown > > No, nothing else is running on that machine. Initially it was working fast > it got slower with that amount of data that in index. Moreover is there a > way to increase buffer size for the Lucene index files (.tim, .doc, and > .pos) from 8K to something much bigger. > > > 1. What JVM version are you running this with? > > java version "1.7.0\_51" > > Java(TM) SE Runtime Environment (build 1.7.0\_51-b13) > > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) > > OS\_NAME="Windows" > > OS\_VERSION="5.2" > > OS\_ARCH="amd64" > > 1. If you changed any of the default settings for merge factors etc - can > > > you revert that and try again? > > Tried before was same behavior. > > > 1. Can you try with embedded=false and see if it makes a difference? > > Tried before was same behavior. > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > On Wed, Apr 9, 2014 at 4:11 PM, Yitzhak Kesselman [ikess...@gmail.com](mailto:ikess...@gmail.com)wrote: > > > > > Hi, > > > > > > I have configured a single node ES with logstash 1.4.0 (8GB memory) with > > > the following configuration: > > > > > > - > > > > > > ## index.number\_of\_shards: 7 > > > > > > ## number\_of\_replicas: 0 > > > > > > ## refresh\_interval: -1 > > > > > > ## translog.flush\_threshold\_ops: 100000 > > > > > > ## merge.policy.merge\_factor: 30 > > > > > > ## codec.bloom.load: false > > > > > > ## min\_shard\_index\_buffer\_size: 12m > > > > > > ## compound\_format : true > > > > > > ## indices.fielddata.cache.size: 15% > > > > > > ## indices.fielddata.cache.expire: 5m > > > > > > ## indices.cache.filter.size: 15% > > > > > > indices.cache.filter.expire: 5m > > > > > > Machine : 16GB RAM, Intel I&-2600 CPU @ 3.4GHZ. > > > OS: 64bit WindowsServer 2012 R2 > > > > > > My raw data is CSV file and I use grok as a filter to parse it with > > > output configuration (elasticsearch { embedded =\> true flush\_size =\> > > > 100000 idle\_flush\_time =\> 30 }). > > > Row data size is about 100GB events per day which ES tries to input into > > > one index (with 7 shards). > > > > > > At the beginning the insert was fast however after a while > > > it's got extremely slow, 1.5K doc in 8K seconds ☹ > > > > > > Currently the index has around 140Million docs with size of 55GB. > > > > > > When I have analyzed the write to the disk with ProcMon I have seen that > > > the process is writing in an interleaved manner to three kinds of files > > > (.tim, .doc, and .pos) in 4K and 8K segments, instead of batching writes to > > > some reasonable number. > > > > > > Appreciate the help. > > > > > > All the best, > > > > > > Yitzhak > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2d4d1ce3-1cf6-47c9-bc24-cdde5235e520%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2d4d1ce3-1cf6-47c9-bc24-cdde5235e520%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yitzhak\_Kesselman](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yitzhak\_Kesselman](https://discuss.elastic.co/u/Yitzhak_Kesselman)\ **Post date:** [April 9, 2014, 7:33pm UTC](https://discuss.elastic.co/t/extremly-slow-troughput-on-large-index/16897/5 "2014-04-09T19:33:28Z")
Attached the index rate (using bigdesk): [https://lh5.googleusercontent.com/-Jve-j75qB9o/U0WgK5ZMvMI/AAAAAAAAAFo/5\_WZuCryeRw/s1600/bigdesk.png](https://lh5.googleusercontent.com/-Jve-j75qB9o/U0WgK5ZMvMI/AAAAAAAAAFo/5_WZuCryeRw/s1600/bigdesk.png) The indexing requests per second is around 2K and the Indexing time per second is around 3K On Wednesday, April 9, 2014 9:36:12 PM UTC+3, Yitzhak Kesselman wrote: > Answers inline. > > Regarding the slow I/O. When I analyzed the creation of the Lucene index > files I see that they are created without any special flags (such as no > buffering or write through). This means that we’re paying costs twice – > when we write the file we’re going cache data in Windows’ Cache Manager, > which takes a lot of memory (which is then not available to the application > itself) but when we read the file we don’t actually read it using the > cache, which makes the operation slow. _Any ideas?_ > > On Wednesday, April 9, 2014 5:28:11 PM UTC+3, Itamar Syn-Hershko wrote: > > > Shooting in the dark here, but here it goes: > > > > 1. Do you have anything else running on the system? for example AVs are > > known to cause slow-downs for such services, and other I/O or memory heavy > > services could cause thrashing or just general slowdown > > No, nothing else is running on that machine. Initially it was working fast > it got slower with that amount of data that in index. Moreover is there a > way to increase buffer size for the Lucene index files (.tim, .doc, and > .pos) from 8K to something much bigger. > > > 1. What JVM version are you running this with? > > java version "1.7.0\_51" > > Java(TM) SE Runtime Environment (build 1.7.0\_51-b13) > > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) > > OS\_NAME="Windows" > > OS\_VERSION="5.2" > > OS\_ARCH="amd64" > > 1. If you changed any of the default settings for merge factors etc - can > > > you revert that and try again? > > Tried before was same behavior. > > > 1. Can you try with embedded=false and see if it makes a difference? > > Tried before was same behavior. > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > On Wed, Apr 9, 2014 at 4:11 PM, Yitzhak Kesselman [ikess...@gmail.com](mailto:ikess...@gmail.com)wrote: > > > > > Hi, > > > > > > I have configured a single node ES with logstash 1.4.0 (8GB memory) with > > > the following configuration: > > > > > > - > > > > > > ## index.number\_of\_shards: 7 > > > > > > ## number\_of\_replicas: 0 > > > > > > ## refresh\_interval: -1 > > > > > > ## translog.flush\_threshold\_ops: 100000 > > > > > > ## merge.policy.merge\_factor: 30 > > > > > > ## codec.bloom.load: false > > > > > > ## min\_shard\_index\_buffer\_size: 12m > > > > > > ## compound\_format : true > > > > > > ## indices.fielddata.cache.size: 15% > > > > > > ## indices.fielddata.cache.expire: 5m > > > > > > ## indices.cache.filter.size: 15% > > > > > > indices.cache.filter.expire: 5m > > > > > > Machine : 16GB RAM, Intel I&-2600 CPU @ 3.4GHZ. > > > OS: 64bit WindowsServer 2012 R2 > > > > > > My raw data is CSV file and I use grok as a filter to parse it with > > > output configuration (elasticsearch { embedded =\> true flush\_size =\> > > > 100000 idle\_flush\_time =\> 30 }). > > > Row data size is about 100GB events per day which ES tries to input into > > > one index (with 7 shards). > > > > > > At the beginning the insert was fast however after a while > > > it's got extremely slow, 1.5K doc in 8K seconds ☹ > > > > > > Currently the index has around 140Million docs with size of 55GB. > > > > > > When I have analyzed the write to the disk with ProcMon I have seen that > > > the process is writing in an interleaved manner to three kinds of files > > > (.tim, .doc, and .pos) in 4K and 8K segments, instead of batching writes to > > > some reasonable number. > > > > > > Appreciate the help. > > > > > > All the best, > > > > > > Yitzhak > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [April 9, 2014, 7:39pm UTC](https://discuss.elastic.co/t/extremly-slow-troughput-on-large-index/16897/6 "2014-04-09T19:39:19Z")
Can you try batching writes to Elasticsearch? see > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Wed, Apr 9, 2014 at 10:33 PM, Yitzhak Kesselman [ikesselman@gmail.com](mailto:ikesselman@gmail.com)wrote: > Attached the index rate (using bigdesk): > > [https://lh5.googleusercontent.com/-Jve-j75qB9o/U0WgK5ZMvMI/AAAAAAAAAFo/5\_WZuCryeRw/s1600/bigdesk.png](https://lh5.googleusercontent.com/-Jve-j75qB9o/U0WgK5ZMvMI/AAAAAAAAAFo/5_WZuCryeRw/s1600/bigdesk.png) > > The indexing requests per second is around 2K and the Indexing time per > second is around 3K > > On Wednesday, April 9, 2014 9:36:12 PM UTC+3, Yitzhak Kesselman wrote: > > > Answers inline. > > > > Regarding the slow I/O. When I analyzed the creation of the Lucene index > > files I see that they are created without any special flags (such as no > > buffering or write through). This means that we're paying costs twice - > > when we write the file we're going cache data in Windows' Cache Manager, > > which takes a lot of memory (which is then not available to the application > > itself) but when we read the file we don't actually read it using the > > cache, which makes the operation slow. _Any ideas?_ > > > > On Wednesday, April 9, 2014 5:28:11 PM UTC+3, Itamar Syn-Hershko wrote: > > > > > Shooting in the dark here, but here it goes: > > > > > > 1. Do you have anything else running on the system? for example AVs are > > > known to cause slow-downs for such services, and other I/O or memory heavy > > > services could cause thrashing or just general slowdown > > > > No, nothing else is running on that machine. Initially it was working > > fast it got slower with that amount of data that in index. Moreover is > > there a way to increase buffer size for the Lucene index files (.tim, .doc, > > and .pos) from 8K to something much bigger. > > > > > 1. What JVM version are you running this with? > > > > java version "1.7.0\_51" > > > > Java(TM) SE Runtime Environment (build 1.7.0\_51-b13) > > > > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) > > > > OS\_NAME="Windows" > > > > OS\_VERSION="5.2" > > > > OS\_ARCH="amd64" > > > > 1. If you changed any of the default settings for merge factors etc - can > > > > > you revert that and try again? > > > > Tried before was same behavior. > > > > > 1. Can you try with embedded=false and see if it makes a difference? > > > > Tried before was same behavior. > > > > > -- > > > > > > Itamar Syn-Hershko > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > Freelance Developer & Consultant > > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > > > On Wed, Apr 9, 2014 at 4:11 PM, Yitzhak Kesselman [ikess...@gmail.com](mailto:ikess...@gmail.com)wrote: > > > > > > > Hi, > > > > > > > > I have configured a single node ES with logstash 1.4.0 > > > > (8GB memory) with the following configuration: > > > > > > > > - > > > > > > > > ## index.number\_of\_shards: 7 > > > > > > > > ## number\_of\_replicas: 0 > > > > > > > > ## refresh\_interval: -1 > > > > > > > > ## translog.flush\_threshold\_ops: 100000 > > > > > > > > ## merge.policy.merge\_factor: 30 > > > > > > > > ## codec.bloom.load: false > > > > > > > > ## min\_shard\_index\_buffer\_size: 12m > > > > > > > > ## compound\_format : true > > > > > > > > ## indices.fielddata.cache.size: 15% > > > > > > > > ## indices.fielddata.cache.expire: 5m > > > > > > > > ## indices.cache.filter.size: 15% > > > > > > > > indices.cache.filter.expire: 5m > > > > > > > > Machine : 16GB RAM, Intel I&-2600 CPU @ 3.4GHZ. > > > > OS: 64bit WindowsServer 2012 R2 > > > > > > > > My raw data is CSV file and I use grok as a filter to parse it with > > > > output configuration (elasticsearch { embedded =\> true flush\_size =\> > > > > 100000 idle\_flush\_time =\> 30 }). > > > > Row data size is about 100GB events per day which ES tries to input > > > > into one index (with 7 shards). > > > > > > > > At the beginning the insert was fast however after a while > > > > it's got extremely slow, 1.5K doc in 8K seconds ☹ > > > > > > > > Currently the index has around 140Million docs with size of 55GB. > > > > > > > > When I have analyzed the write to the disk with ProcMon I have seen > > > > that the process is writing in an interleaved manner to three kinds of > > > > files (.tim, .doc, and .pos) in 4K and 8K segments, instead of batching > > > > writes to some reasonable number. > > > > > > > > Appreciate the help. > > > > > > > > All the best, > > > > > > > > Yitzhak > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8% > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvQcxHRMbZdf%3Di1-EmoP1Dq3BxKW%2Bau0r0Ox8fbJSRo%2Bw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvQcxHRMbZdf%3Di1-EmoP1Dq3BxKW%2Bau0r0Ox8fbJSRo%2Bw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yitzhak\_Kesselman](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yitzhak\_Kesselman](https://discuss.elastic.co/u/Yitzhak_Kesselman)\ **Post date:** [April 10, 2014, 5:13am UTC](https://discuss.elastic.co/t/extremly-slow-troughput-on-large-index/16897/7 "2014-04-10T05:13:33Z")
I am using logstash 1.4.0, if I had understand correctly it uses automatically the Bulk API. Do I miss something ? Is there a limit on the size of an Index (on single node machine)? (BTW Itamar thanks for the help!) On Wednesday, April 9, 2014 10:39:19 PM UTC+3, Itamar Syn-Hershko wrote: > Can you try batching writes to Elasticsearch? see > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html) > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > On Wed, Apr 9, 2014 at 10:33 PM, Yitzhak Kesselman \<[ikess...@gmail.com](mailto:ikess...@gmail.com)\ > > > wrote: > > > Attached the index rate (using bigdesk): > > [https://lh5.googleusercontent.com/-Jve-j75qB9o/U0WgK5ZMvMI/AAAAAAAAAFo/5\_WZuCryeRw/s1600/bigdesk.png](https://lh5.googleusercontent.com/-Jve-j75qB9o/U0WgK5ZMvMI/AAAAAAAAAFo/5_WZuCryeRw/s1600/bigdesk.png) > > > > The indexing requests per second is around 2K and the Indexing time per > > second is around 3K > > > > On Wednesday, April 9, 2014 9:36:12 PM UTC+3, Yitzhak Kesselman wrote: > > > > > Answers inline. > > > > > > Regarding the slow I/O. When I analyzed the creation of the Lucene > > > index files I see that they are created without any special flags (such as > > > no buffering or write through). This means that we’re paying costs twice – > > > when we write the file we’re going cache data in Windows’ Cache Manager, > > > which takes a lot of memory (which is then not available to the application > > > itself) but when we read the file we don’t actually read it using the > > > cache, which makes the operation slow. _Any ideas?_ > > > > > > On Wednesday, April 9, 2014 5:28:11 PM UTC+3, Itamar Syn-Hershko wrote: > > > > > > > Shooting in the dark here, but here it goes: > > > > > > > > 1. Do you have anything else running on the system? for example AVs are > > > > known to cause slow-downs for such services, and other I/O or memory heavy > > > > services could cause thrashing or just general slowdown > > > > > > No, nothing else is running on that machine. Initially it was working > > > fast it got slower with that amount of data that in index. Moreover is > > > there a way to increase buffer size for the Lucene index files (.tim, .doc, > > > and .pos) from 8K to something much bigger. > > > > > > > 1. What JVM version are you running this with? > > > > > > java version "1.7.0\_51" > > > > > > Java(TM) SE Runtime Environment (build 1.7.0\_51-b13) > > > > > > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) > > > > > > OS\_NAME="Windows" > > > > > > OS\_VERSION="5.2" > > > > > > OS\_ARCH="amd64" > > > > > > 1. If you changed any of the default settings for merge factors etc - > > > > > > > can you revert that and try again? > > > > > > Tried before was same behavior. > > > > > > > 1. Can you try with embedded=false and see if it makes a difference? > > > > > > Tried before was same behavior. > > > > > > > -- > > > > > > > > Itamar Syn-Hershko > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > Freelance Developer & Consultant > > > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > > > > > On Wed, Apr 9, 2014 at 4:11 PM, Yitzhak Kesselman [ikess...@gmail.com](mailto:ikess...@gmail.com)wrote: > > > > > > > > > Hi, > > > > > > > > > > I have configured a single node ES with logstash 1.4.0 > > > > > (8GB memory) with the following configuration: > > > > > > > > > > - > > > > > > > > > > ## index.number\_of\_shards: 7 > > > > > > > > > > ## number\_of\_replicas: 0 > > > > > > > > > > ## refresh\_interval: -1 > > > > > > > > > > ## translog.flush\_threshold\_ops: 100000 > > > > > > > > > > ## merge.policy.merge\_factor: 30 > > > > > > > > > > ## codec.bloom.load: false > > > > > > > > > > ## min\_shard\_index\_buffer\_size: 12m > > > > > > > > > > ## compound\_format : true > > > > > > > > > > ## indices.fielddata.cache.size: 15% > > > > > > > > > > ## indices.fielddata.cache.expire: 5m > > > > > > > > > > ## indices.cache.filter.size: 15% > > > > > > > > > > indices.cache.filter.expire: 5m > > > > > > > > > > Machine : 16GB RAM, Intel I&-2600 CPU @ 3.4GHZ. > > > > > OS: 64bit WindowsServer 2012 R2 > > > > > > > > > > My raw data is CSV file and I use grok as a filter to parse it with > > > > > output configuration (elasticsearch { embedded =\> true flush\_size =\> > > > > > 100000 idle\_flush\_time =\> 30 }). > > > > > Row data size is about 100GB events per day which ES tries to input > > > > > into one index (with 7 shards). > > > > > > > > > > At the beginning the insert was fast however after a while > > > > > it's got extremely slow, 1.5K doc in 8K seconds ☹ > > > > > > > > > > Currently the index has around 140Million docs with size of 55GB. > > > > > > > > > > When I have analyzed the write to the disk with ProcMon I have seen > > > > > that the process is writing in an interleaved manner to three kinds of > > > > > files (.tim, .doc, and .pos) in 4K and 8K segments, instead of batching > > > > > writes to some reasonable number. > > > > > > > > > > Appreciate the help. > > > > > > > > > > All the best, > > > > > > > > > > Yitzhak > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8% > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1a97a59e-aeec-4b3a-bbb7-5e3df09ed3ec%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1a97a59e-aeec-4b3a-bbb7-5e3df09ed3ec%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yitzhak\_Kesselman](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yitzhak\_Kesselman](https://discuss.elastic.co/u/Yitzhak_Kesselman)\ **Post date:** [April 22, 2014, 1:29pm UTC](https://discuss.elastic.co/t/extremly-slow-troughput-on-large-index/16897/8 "2014-04-22T13:29:44Z")
Pushing up... On Thursday, April 10, 2014 8:13:33 AM UTC+3, Yitzhak Kesselman wrote: > I am using logstash 1.4.0, if I had understand correctly it uses > automatically the Bulk API. Do I miss something ? > Is there a limit on the size of an Index (on single node machine)? > > (BTW Itamar thanks for the help!) > > On Wednesday, April 9, 2014 10:39:19 PM UTC+3, Itamar Syn-Hershko wrote: > > > Can you try batching writes to Elasticsearch? see > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html) > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > On Wed, Apr 9, 2014 at 10:33 PM, Yitzhak Kesselman [ikess...@gmail.com](mailto:ikess...@gmail.com)wrote: > > > > > Attached the index rate (using bigdesk): > > > [https://lh5.googleusercontent.com/-Jve-j75qB9o/U0WgK5ZMvMI/AAAAAAAAAFo/5\_WZuCryeRw/s1600/bigdesk.png](https://lh5.googleusercontent.com/-Jve-j75qB9o/U0WgK5ZMvMI/AAAAAAAAAFo/5_WZuCryeRw/s1600/bigdesk.png) > > > > > > The indexing requests per second is around 2K and the Indexing time per > > > second is around 3K > > > > > > On Wednesday, April 9, 2014 9:36:12 PM UTC+3, Yitzhak Kesselman wrote: > > > > > > > Answers inline. > > > > > > > > Regarding the slow I/O. When I analyzed the creation of the Lucene > > > > index files I see that they are created without any special flags (such as > > > > no buffering or write through). This means that we’re paying costs twice – > > > > when we write the file we’re going cache data in Windows’ Cache Manager, > > > > which takes a lot of memory (which is then not available to the application > > > > itself) but when we read the file we don’t actually read it using the > > > > cache, which makes the operation slow. _Any ideas?_ > > > > > > > > On Wednesday, April 9, 2014 5:28:11 PM UTC+3, Itamar Syn-Hershko wrote: > > > > > > > > > Shooting in the dark here, but here it goes: > > > > > > > > > > 1. Do you have anything else running on the system? for example AVs > > > > > are known to cause slow-downs for such services, and other I/O or memory > > > > > heavy services could cause thrashing or just general slowdown > > > > > > > > No, nothing else is running on that machine. Initially it was working > > > > fast it got slower with that amount of data that in index. Moreover is > > > > there a way to increase buffer size for the Lucene index files (.tim, .doc, > > > > and .pos) from 8K to something much bigger. > > > > > > > > > 1. What JVM version are you running this with? > > > > > > > > java version "1.7.0\_51" > > > > > > > > Java(TM) SE Runtime Environment (build 1.7.0\_51-b13) > > > > > > > > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) > > > > > > > > OS\_NAME="Windows" > > > > > > > > OS\_VERSION="5.2" > > > > > > > > OS\_ARCH="amd64" > > > > > > > > 1. If you changed any of the default settings for merge factors etc - > > > > > > > > > can you revert that and try again? > > > > > > > > Tried before was same behavior. > > > > > > > > > 1. Can you try with embedded=false and see if it makes a difference? > > > > > > > > Tried before was same behavior. > > > > > > > > > -- > > > > > > > > > > Itamar Syn-Hershko > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > Freelance Developer & Consultant > > > > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > > > > > > > On Wed, Apr 9, 2014 at 4:11 PM, Yitzhak Kesselman [ikess...@gmail.com](mailto:ikess...@gmail.com)wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I have configured a single node ES with logstash 1.4.0 > > > > > > (8GB memory) with the following configuration: > > > > > > > > > > > > - > > > > > > > > > > > > ## index.number\_of\_shards: 7 > > > > > > > > > > > > ## number\_of\_replicas: 0 > > > > > > > > > > > > ## refresh\_interval: -1 > > > > > > > > > > > > ## translog.flush\_threshold\_ops: 100000 > > > > > > > > > > > > ## merge.policy.merge\_factor: 30 > > > > > > > > > > > > ## codec.bloom.load: false > > > > > > > > > > > > ## min\_shard\_index\_buffer\_size: 12m > > > > > > > > > > > > ## compound\_format : true > > > > > > > > > > > > ## indices.fielddata.cache.size: 15% > > > > > > > > > > > > ## indices.fielddata.cache.expire: 5m > > > > > > > > > > > > ## indices.cache.filter.size: 15% > > > > > > > > > > > > indices.cache.filter.expire: 5m > > > > > > > > > > > > Machine : 16GB RAM, Intel I&-2600 CPU @ 3.4GHZ. > > > > > > OS: 64bit WindowsServer 2012 R2 > > > > > > > > > > > > My raw data is CSV file and I use grok as a filter to parse it with > > > > > > output configuration (elasticsearch { embedded =\> true flush\_size =\> > > > > > > 100000 idle\_flush\_time =\> 30 }). > > > > > > Row data size is about 100GB events per day which ES tries to input > > > > > > into one index (with 7 shards). > > > > > > > > > > > > At the beginning the insert was fast however after a while > > > > > > it's got extremely slow, 1.5K doc in 8K seconds ☹ > > > > > > > > > > > > Currently the index has around 140Million docs with size of 55GB. > > > > > > > > > > > > When I have analyzed the write to the disk with ProcMon I have seen > > > > > > that the process is writing in an interleaved manner to three kinds of > > > > > > files (.tim, .doc, and .pos) in 4K and 8K segments, instead of batching > > > > > > writes to some reasonable number. > > > > > > > > > > > > Appreciate the help. > > > > > > > > > > > > All the best, > > > > > > > > > > > > Yitzhak > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8% > > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aa0193d2-cb6a-49e4-b4a3-a2f821f732f8%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6c6d0aa9-110e-4e6a-9aa2-de0d5b487c6a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/065713b1-fb28-4a72-9a17-ac6b1e5426b0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/065713b1-fb28-4a72-9a17-ac6b1e5426b0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:34am UTC](https://discuss.elastic.co/t/extremly-slow-troughput-on-large-index/16897/9 "2017-07-06T01:34:21Z")