# ES searching and indexing speed reduced after processing 600milion records **URL:** \ **Category:** Elasticsearch\ **Created:** [March 6, 2013, 3:40pm UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035 "2013-03-06T15:40:50Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [March 6, 2013, 3:40pm UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/1 "2013-03-06T15:40:50Z")
Hi all, We have a cluster with 16 nodes. Total memory is 32g for each server. On each node, ES and 3 other java process are running. Assigned memory to ES is 12g(xms and xmx are equal), set 7g for other java processes and 9g left for the OS. There are 2 big index. Application read from one, process and store into another index. the source index has about 1.1 billion records and final data in destination index also would be almost same(1.1billion). So at the same time we have read and write. At first when the destination index was empty the application speed was acceptable(read, process and write). but after reaching to 600 milion data in destination index, Es crashed with OOM error. Now I increased the Es memory to 16g but the speed reduced almost to half. each node at same time load 100k records, processes and store them. each document size is about .5k after ES compression. ES configuration: bootstrap.mlockall: true index.number\_of\_shards: 51 index.number\_of\_replicas: 1 index.store.compress.stored, true index.routing.allocation.total\_shards\_per\_node, 7 index.merge.policy.use\_compound\_file, false index.compound\_format, false indices.memory.index\_buffer\_size, 20% index.translog.flush\_threshold\_period, 60s index.merge.policy.merge\_factor, 30 index.store.throttle.type, merge index.store.throttle.max\_bytes\_per\_sec, 5mb index.cache.filter.max\_size, 10 index.cache.filter.expire, 1m index.cache.field.type, resident index.cache.field.expire, 1m So I want to know if it is ES limitation and after reaching such amount of data we need to create a new index or we have missed some things. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 6, 2013, 6:24pm UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/2 "2013-03-06T18:24:13Z")
Hiya Vahid You are retrieving and indexing 100k records at a time? That sounds like a lot. Don't forget that all of that data needs to be present in memory at the same time. In my experiments, I usually found the sweetspot to be 1,000 - 5,000 records at a time. Results will vary by the type of data, etc, but I'd try just reducing the bulk size. Clint On Wed, Mar 6, 2013 at 4:40 PM, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) wrote: > Hi all, > > We have a cluster with 16 nodes. > Total memory is 32g for each server. On each node, ES and 3 other java > process are running. Assigned memory to ES is 12g(xms and xmx are equal), > set 7g for other java processes and 9g left for the OS. > There are 2 big index. Application read from one, process and store into > another index. the source index has about 1.1 billion records and final > data in destination index also would be almost same(1.1billion). So at the > same time we have read and write. At first when the destination index was > empty the application speed was acceptable(read, process and write). > but after reaching to 600 milion data in destination index, Es crashed > with OOM error. Now I increased the Es memory to 16g but the speed reduced > almost to half. > each node at same time load 100k records, processes and store them. > each document size is about .5k after ES compression. > ES configuration: > > bootstrap.mlockall: true > index.number\_of\_shards: 51 > index.number\_of\_replicas: 1 > index.store.compress.stored, true > index.routing.allocation.total\_shards\_per\_node, 7 > index.merge.policy.use\_compound\_file, false > index.compound\_format, false > indices.memory.index\_buffer\_size, 20% > index.translog.flush\_threshold\_period, 60s > index.merge.policy.merge\_factor, 30 > index.store.throttle.type, merge > index.store.throttle.max\_bytes\_per\_sec, 5mb > index.cache.filter.max\_size, 10 > index.cache.filter.expire, 1m > index.cache.field.type, resident > index.cache.field.expire, 1m > > So I want to know if it is ES limitation and after reaching such amount of > data we need to create a new index or we have missed some things. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 6, 2013, 7:46pm UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/3 "2013-03-06T19:46:32Z")
I don't have insight into your application how it works and how much memory it uses. But you should be aware that building an index of about 1.1 billion documents while using an extra application on the same machines will use significant more memory than just inserting documents without extra processing. Also I don't know anything about your method of processing, if you process documents in batches, if it's inside or outside ES, if it's Java or not. This all may add up to resource consumption in many ways. My personal impression - without knowledge of the nature of your data processing - is that with reorganizing memory consumption you should be able to streamline the processing. Take care of your algorithm, reduce the memory footprint. Do not rely only on modifying the heap size parameter, it does not change much of the nature of your task. For example, reducing the result set size and scrolling over small portions of a result set may contribute to reducing the memory footprint. If your algorithm is really bad, the heap could be as large as your machine allows and it will still fail. If you ask me, 7G on heap, this is so huge, it should be sufficient for even the most demanding stream processing applications. Jörg Am 06.03.13 16:40, schrieb Vahid: > Hi all, > > We have a cluster with 16 nodes. > Total memory is 32g for each server. On each node, ES and 3 other java > process are running. Assigned memory to ES is 12g(xms and xmx are > equal), set 7g for other java processes and 9g left for the OS. > There are 2 big index. Application read from one, process and store > into another index. the source index has about 1.1 billion records and > final data in destination index also would be almost same(1.1billion). > So at the same time we have read and write. At first when the > destination index was empty the application speed was acceptable(read, > process and write). > but after reaching to 600 milion data in destination index, Es crashed > with OOM error. Now I increased the Es memory to 16g but the speed > reduced almost to half. > each node at same time load 100k records, processes and store them. > each document size is about .5k after ES compression. > ES configuration: > > bootstrap.mlockall: true > index.number\_of\_shards: 51 > index.number\_of\_replicas: 1 > index.store.compress.stored, true > index.routing.allocation.total\_shards\_per\_node, 7 > index.merge.policy.use\_compound\_file, false > index.compound\_format, false > indices.memory.index\_buffer\_size, 20% > index.translog.flush\_threshold\_period, 60s > index.merge.policy.merge\_factor, 30 > index.store.throttle.type, merge > index.store.throttle.max\_bytes\_per\_sec, 5mb > index.cache.filter.max\_size, 10 > index.cache.filter.expire, 1m > index.cache.field.type, resident > index.cache.field.expire, 1m > > So I want to know if it is ES limitation and after reaching such > amount of data we need to create a new index or we have missed some > things. > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [March 7, 2013, 11:05am UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/4 "2013-03-07T11:05:27Z")
Thank you both Clinton and Jörg for your replies. Sorry for bad explanation. I'll try again: All the applications are developed by java. ES just stores the data and processing done outside the ES. Let say we have a manager application named BDM, source index s1 and destination index s2. 1- On each node BDM sends a query to ES and (16 is No. of the nodes) load 100k records from s1 index (means total loaded data by all the nodes are 1.6m records). 2- BDMs process the fetched data and send bulk inserts to s2 index on ES(100k bulk size). 3- BDMs send bulk update (100k) to s1 index on ES. Assigned memory to BDM is 5g and it has no problem with loading and processing the data. Size of the data for 100k document is about 400mb. 16g of memory assigned to ES on each node. Number of records on s1 is 1.1 billion. At start up s2 was empty and speed was ok for us and assigned memory to ES was 11g. After processing 600m records and storing them into s2 we got oom error and I've increased the memory to 16g. Now after running the jobs for let say 5 hours, split brain occurred and I observed in the ES logs on some nodes warning for GC and after that node got out of the cluster and I had to restart the node. On the bigdesk I see that cache size is about 100mb and during the loading and indexig the data heap old generation size is about 70% of hole the assigned memory. So I don't know why ES uses such huge size of memory that causes to split brain and also performance problems. Thanks again for your helps, Vahid On Wednesday, March 6, 2013 8:46:32 PM UTC+1, Jörg Prante wrote: > I don't have insight into your application how it works and how much > memory it uses. But you should be aware that building an index of about > 1.1 billion documents while using an extra application on the same > machines will use significant more memory than just inserting documents > without extra processing. Also I don't know anything about your method > of processing, if you process documents in batches, if it's inside or > outside ES, if it's Java or not. This all may add up to resource > consumption in many ways. > > My personal impression - without knowledge of the nature of your data > processing - is that with reorganizing memory consumption you should be > able to streamline the processing. Take care of your algorithm, reduce > the memory footprint. Do not rely only on modifying the heap size > parameter, it does not change much of the nature of your task. For > example, reducing the result set size and scrolling over small portions > of a result set may contribute to reducing the memory footprint. If your > algorithm is really bad, the heap could be as large as your machine > allows and it will still fail. If you ask me, 7G on heap, this is so > huge, it should be sufficient for even the most demanding stream > processing applications. > > Jörg > > Am 06.03.13 16:40, schrieb Vahid: > > > Hi all, > > > > We have a cluster with 16 nodes. > > Total memory is 32g for each server. On each node, ES and 3 other java > > process are running. Assigned memory to ES is 12g(xms and xmx are > > equal), set 7g for other java processes and 9g left for the OS. > > There are 2 big index. Application read from one, process and store > > into another index. the source index has about 1.1 billion records and > > final data in destination index also would be almost same(1.1billion). > > So at the same time we have read and write. At first when the > > destination index was empty the application speed was acceptable(read, > > process and write). > > but after reaching to 600 milion data in destination index, Es crashed > > with OOM error. Now I increased the Es memory to 16g but the speed > > reduced almost to half. > > each node at same time load 100k records, processes and store them. > > each document size is about .5k after ES compression. > > ES configuration: > > > > bootstrap.mlockall: true > > index.number\_of\_shards: 51 > > index.number\_of\_replicas: 1 > > index.store.compress.stored, true > > index.routing.allocation.total\_shards\_per\_node, 7 > > index.merge.policy.use\_compound\_file, false > > index.compound\_format, false > > indices.memory.index\_buffer\_size, 20% > > index.translog.flush\_threshold\_period, 60s > > index.merge.policy.merge\_factor, 30 > > index.store.throttle.type, merge > > index.store.throttle.max\_bytes\_per\_sec, 5mb > > index.cache.filter.max\_size, 10 > > index.cache.filter.expire, 1m > > index.cache.field.type, resident > > index.cache.field.expire, 1m > > > > So I want to know if it is ES limitation and after reaching such > > amount of data we need to create a new index or we have missed some > > things. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 7, 2013, 12:35pm UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/5 "2013-03-07T12:35:54Z")
You overload the system. My recommendations for standard hardware: - use smaller bulk request. 100k docs = 400mb is too much. Around 1000 docs = 4mb is ok. - measure the response time. It should be around 10-100 milliseconds. As the index gets bigger, response time will increase but only slowly. Not much more than a second. - delays in the response are due to the indexing load of the cluster - run bulk requests in parallel, but throttle at a certain number of concurrency. Say 30. Wait for bulk response if you reach the concurrecny limit before continue. - advanced setting is to adjust the concurrency rate to the available CPU power, but YMMV With this approach, and just 1g of heap, I can index millions and millions of docs in the standard(!) setting, constantly running for hours and hours. You can add more heap, but it will not change much regarding performance. Split brain is only a side effect. What you see is response timeout of nodes because you overloaded them. The master waits at max 5s and a node must respond. Jörg Am 07.03.2013 12:05, schrieb Vahid: > Thank you both Clinton and Jörg for your replies. > Sorry for bad explanation. I'll try again: > > All the applications are developed by java. ES just stores the data > and processing done outside the ES. > Let say we have a manager application named BDM, source index s1 and > destination index s2. > 1- On each node BDM sends a query to ES and (16 is No. of the nodes) > load 100k records from s1 index (means total loaded data by all the > nodes are 1.6m records). > 2- BDMs process the fetched data and send bulk inserts to s2 index on > ES(100k bulk size). > 3- BDMs send bulk update (100k) to s1 index on ES. > Assigned memory to BDM is 5g and it has no problem with loading and > processing the data. > Size of the data for 100k document is about 400mb. 16g of memory > assigned to ES on each node. > > Number of records on s1 is 1.1 billion. At start up s2 was empty and > speed was ok for us and assigned memory to ES was 11g. > After processing 600m records and storing them into s2 we got oom > error and I've increased the memory to 16g. > > Now after running the jobs for let say 5 hours, split brain occurred > and I observed in the ES logs on some nodes warning for GC and after > that node got out of the cluster and I had to restart the node. > On the bigdesk I see that cache size is about 100mb and during the > loading and indexig the data heap old generation size is about 70% of > hole the assigned memory. > > So I don't know why ES uses such huge size of memory that causes to > split brain and also performance problems. > > Thanks again for your helps, > Vahid > > On Wednesday, March 6, 2013 8:46:32 PM UTC+1, Jörg Prante wrote: > > ``` > I don't have insight into your application how it works and how much > memory it uses. But you should be aware that building an index of > about > 1.1 billion documents while using an extra application on the same > machines will use significant more memory than just inserting > documents > without extra processing. Also I don't know anything about your > method > of processing, if you process documents in batches, if it's inside or > outside ES, if it's Java or not. This all may add up to resource > consumption in many ways. > > My personal impression - without knowledge of the nature of your data > processing - is that with reorganizing memory consumption you > should be > able to streamline the processing. Take care of your algorithm, > reduce > the memory footprint. Do not rely only on modifying the heap size > parameter, it does not change much of the nature of your task. For > example, reducing the result set size and scrolling over small > portions > of a result set may contribute to reducing the memory footprint. > If your > algorithm is really bad, the heap could be as large as your machine > allows and it will still fail. If you ask me, 7G on heap, this is so > huge, it should be sufficient for even the most demanding stream > processing applications. > > Jörg > > Am 06.03.13 16:40, schrieb Vahid: > > Hi all, > > > > We have a cluster with 16 nodes. > > Total memory is 32g for each server. On each node, ES and 3 > other java > > process are running. Assigned memory to ES is 12g(xms and xmx are > > equal), set 7g for other java processes and 9g left for the OS. > > There are 2 big index. Application read from one, process and store > > into another index. the source index has about 1.1 billion > records and > > final data in destination index also would be almost > same(1.1billion). > > So at the same time we have read and write. At first when the > > destination index was empty the application speed was > acceptable(read, > > process and write). > > but after reaching to 600 milion data in destination index, Es > crashed > > with OOM error. Now I increased the Es memory to 16g but the speed > > reduced almost to half. > > each node at same time load 100k records, processes and store them. > > each document size is about .5k after ES compression. > > ES configuration: > > > > bootstrap.mlockall: true > > index.number_of_shards: 51 > > index.number_of_replicas: 1 > > index.store.compress.stored, true > > index.routing.allocation.total_shards_per_node, 7 > > index.merge.policy.use_compound_file, false > > index.compound_format, false > > indices.memory.index_buffer_size, 20% > > index.translog.flush_threshold_period, 60s > > index.merge.policy.merge_factor, 30 > > index.store.throttle.type, merge > > index.store.throttle.max_bytes_per_sec, 5mb > > index.cache.filter.max_size, 10 > > index.cache.filter.expire, 1m > > index.cache.field.type, resident > > index.cache.field.expire, 1m > > > > So I want to know if it is ES limitation and after reaching such > > amount of data we need to create a new index or we have missed some > > things. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from > it, send > > an email to elasticsearc...@googlegroups.com. > > For more options, visit https://groups.google.com/groups/opt_out > . > > > > > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [March 7, 2013, 2:49pm UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/6 "2013-03-07T14:49:47Z")
Hey Jörg, Thanks a lot for your great response! I do agree with you that maybe we overloaded the system during the save process. Loading (searching) and indexing in our application are two different phases and they executed sequentially. So I'm wondering why the search speed decreased so much?! Anyway I will change our indexing strategy and hope that searching affected too. Best, Vahid On Thursday, March 7, 2013 1:35:54 PM UTC+1, Jörg Prante wrote: > You overload the system. My recommendations for standard hardware: > > - use smaller bulk request. 100k docs = 400mb is too much. Around 1000 > docs = 4mb is ok. > - measure the response time. It should be around 10-100 milliseconds. As > the index gets bigger, response time will increase but only slowly. Not > much more than a second. > - delays in the response are due to the indexing load of the cluster > - run bulk requests in parallel, but throttle at a certain number of > concurrency. Say 30. Wait for bulk response if you reach the concurrecny > limit before continue. > - advanced setting is to adjust the concurrency rate to the available > CPU power, but YMMV > > With this approach, and just 1g of heap, I can index millions and > millions of docs in the standard(!) setting, constantly running for > hours and hours. You can add more heap, but it will not change much > regarding performance. > > Split brain is only a side effect. What you see is response timeout of > nodes because you overloaded them. The master waits at max 5s and a node > must respond. > > Jörg > > Am 07.03.2013 12:05, schrieb Vahid: > > > Thank you both Clinton and Jörg for your replies. > > Sorry for bad explanation. I'll try again: > > > > All the applications are developed by java. ES just stores the data > > and processing done outside the ES. > > Let say we have a manager application named BDM, source index s1 and > > destination index s2. > > 1- On each node BDM sends a query to ES and (16 is No. of the nodes) > > load 100k records from s1 index (means total loaded data by all the > > nodes are 1.6m records). > > 2- BDMs process the fetched data and send bulk inserts to s2 index on > > ES(100k bulk size). > > 3- BDMs send bulk update (100k) to s1 index on ES. > > Assigned memory to BDM is 5g and it has no problem with loading and > > processing the data. > > Size of the data for 100k document is about 400mb. 16g of memory > > assigned to ES on each node. > > > > Number of records on s1 is 1.1 billion. At start up s2 was empty and > > speed was ok for us and assigned memory to ES was 11g. > > After processing 600m records and storing them into s2 we got oom > > error and I've increased the memory to 16g. > > > > Now after running the jobs for let say 5 hours, split brain occurred > > and I observed in the ES logs on some nodes warning for GC and after > > that node got out of the cluster and I had to restart the node. > > On the bigdesk I see that cache size is about 100mb and during the > > loading and indexig the data heap old generation size is about 70% of > > hole the assigned memory. > > > > So I don't know why ES uses such huge size of memory that causes to > > split brain and also performance problems. > > > > Thanks again for your helps, > > Vahid > > > > On Wednesday, March 6, 2013 8:46:32 PM UTC+1, Jörg Prante wrote: > > > > ``` > > I don't have insight into your application how it works and how much > > memory it uses. But you should be aware that building an index of > > about > > 1.1 billion documents while using an extra application on the same > > machines will use significant more memory than just inserting > > documents > > without extra processing. Also I don't know anything about your > > method > > of processing, if you process documents in batches, if it's inside > > > > ``` > > or > > > ``` > > outside ES, if it's Java or not. This all may add up to resource > > consumption in many ways. > > > > My personal impression - without knowledge of the nature of your > > > > ``` > > data > > > ``` > > processing - is that with reorganizing memory consumption you > > should be > > able to streamline the processing. Take care of your algorithm, > > reduce > > the memory footprint. Do not rely only on modifying the heap size > > parameter, it does not change much of the nature of your task. For > > example, reducing the result set size and scrolling over small > > portions > > of a result set may contribute to reducing the memory footprint. > > If your > > algorithm is really bad, the heap could be as large as your machine > > allows and it will still fail. If you ask me, 7G on heap, this is so > > huge, it should be sufficient for even the most demanding stream > > processing applications. > > > > Jörg > > > > Am 06.03.13 16:40, schrieb Vahid: > > > Hi all, > > > > > > We have a cluster with 16 nodes. > > > Total memory is 32g for each server. On each node, ES and 3 > > other java > > > process are running. Assigned memory to ES is 12g(xms and xmx are > > > equal), set 7g for other java processes and 9g left for the OS. > > > There are 2 big index. Application read from one, process and > > > > ``` > > store > > > ``` > > > into another index. the source index has about 1.1 billion > > records and > > > final data in destination index also would be almost > > same(1.1billion). > > > So at the same time we have read and write. At first when the > > > destination index was empty the application speed was > > acceptable(read, > > > process and write). > > > but after reaching to 600 milion data in destination index, Es > > crashed > > > with OOM error. Now I increased the Es memory to 16g but the speed > > > reduced almost to half. > > > each node at same time load 100k records, processes and store > > > > ``` > > them. > > > ``` > > > each document size is about .5k after ES compression. > > > ES configuration: > > > > > > bootstrap.mlockall: true > > > index.number_of_shards: 51 > > > index.number_of_replicas: 1 > > > index.store.compress.stored, true > > > index.routing.allocation.total_shards_per_node, 7 > > > index.merge.policy.use_compound_file, false > > > index.compound_format, false > > > indices.memory.index_buffer_size, 20% > > > index.translog.flush_threshold_period, 60s > > > index.merge.policy.merge_factor, 30 > > > index.store.throttle.type, merge > > > index.store.throttle.max_bytes_per_sec, 5mb > > > index.cache.filter.max_size, 10 > > > index.cache.filter.expire, 1m > > > index.cache.field.type, resident > > > index.cache.field.expire, 1m > > > > > > So I want to know if it is ES limitation and after reaching such > > > amount of data we need to create a new index or we have missed > > > > ``` > > some > > > ``` > > > things. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from > > it, send > > > an email to elasticsearc...@googlegroups.com. > > > For more options, visit https://groups.google.com/groups/opt_out > > . > > > > > > > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 8, 2013, 9:07am UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/7 "2013-03-08T09:07:43Z")
Maybe I missed it. What is your search speed? What do your searches look like, how many hits do you create? I think, in the case you ask ES to create large result sets, you get some extra peaks on the heap, so I assume a search/scroll strategy and smaller result sets could help then. Jörg Am 07.03.13 15:49, schrieb Vahid: > So I'm wondering why the search speed decreased so much?! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [March 8, 2013, 10:54am UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/8 "2013-03-08T10:54:38Z")
We already have tested different strategies. At first we had multiple queries and result sets(using no filter), as it was not enough fast we changed it into one query using filters. We don't use ES paging and each node fetch 100k document at once without scrolling. At first it was very good and increased our speed 4 time, but after one day searching and indexing we got into memory problems and also speed decreased to half. Also system is not stable and after some hours I can see GC warning messages and ES split brain or no node available exception. Any way I'll try again different scenarios. Best, Vahid On Friday, March 8, 2013 10:07:43 AM UTC+1, Jörg Prante wrote: > Maybe I missed it. What is your search speed? What do your searches look > like, how many hits do you create? I think, in the case you ask ES to > create large result sets, you get some extra peaks on the heap, so I > assume a search/scroll strategy and smaller result sets could help then. > > Jörg > > Am 07.03.13 15:49, schrieb Vahid: > > > So I'm wondering why the search speed decreased so much?! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [March 8, 2013, 1:50pm UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/9 "2013-03-08T13:50:14Z")
I would say your strategy of requesting 100k documents has a non-negligible overhead, which will continue to grow as your index grows. In the default search execution (query\_then\_fetch), you are effectively asking ES to do this for every query: - For each \*shard \*in the index: - Find 100,000 results that match this query - Sort based on score (and potentially custom scores) - Return doc IDs and scores to the requesting node - Then, merge this list of (100,000 \* #Shards), resort based on score (and potentially custom scores) - Take the top 100,000 queries from this resorted list, request individual docs from each relevant shard - Merge results into a set, return to client Notice how ES has to ask each shard (either a primary or replica) for the entire 100k result set. ES has no way of knowing where the results actually lives - the top N results may all reside on a single shard. As such, ES has to query each shard for the full result set. In a 5 shard setup, this means the requesting node will need to merge and sort 500k results to identify the top 100k. Throw in network latency of actually transferring 100k documents from each shard to the requesting node (and then the client), things start to slow down. If you do custom sorts or facets, that adds even more memory required since the appropriate fields need to be fully in memory for sorting. On Friday, March 8, 2013 5:54:38 AM UTC-5, Vahid wrote: > We already have tested different strategies. At first we had multiple > queries and result sets(using no filter), as it was not enough fast we > changed it into one query using filters. We don't use ES paging and each > node fetch 100k document at once without scrolling. > At first it was very good and increased our speed 4 time, but after one > day searching and indexing we got into memory problems and also speed > decreased to half. Also system is not stable and after some hours I can see > GC warning messages and ES split brain or no node available exception. > Any way I'll try again different scenarios. > > Best, > Vahid > > On Friday, March 8, 2013 10:07:43 AM UTC+1, Jörg Prante wrote: > > > Maybe I missed it. What is your search speed? What do your searches look > > like, how many hits do you create? I think, in the case you ask ES to > > create large result sets, you get some extra peaks on the heap, so I > > assume a search/scroll strategy and smaller result sets could help then. > > > > Jörg > > > > Am 07.03.13 15:49, schrieb Vahid: > > > > > So I'm wondering why the search speed decreased so much?! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [March 8, 2013, 3:05pm UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/10 "2013-03-08T15:05:06Z")
Hi Zachary, Thank you for the good points, The index which I'm searching doesn't grow up and the data on that just get updated. At start up each node was loading 100k document at 40 seconds which means by 16 nodes we were loading 1.6m documents within 40 seconds and it was an acceptable performace for us. As we don't need sorting and scoring so mybe I need to disable the sorting and scoring which I think is done internally by ES. I hope this also help to have a more stable search speed. Best, Vahid On Friday, March 8, 2013 2:50:14 PM UTC+1, Zachary Tong wrote: > I would say your strategy of requesting 100k documents has a > non-negligible overhead, which will continue to grow as your index grows. > In the default search execution (query\_then\_fetch), you are effectively > asking ES to do this for every query: > > - For each \*shard \*in the index: > - Find 100,000 results that match this query > - Sort based on score (and potentially custom scores) > - Return doc IDs and scores to the requesting node > > - Then, merge this list of (100,000 \* #Shards), resort based on score > (and potentially custom scores) > - Take the top 100,000 queries from this resorted list, request > individual docs from each relevant shard > - Merge results into a set, return to client > > Notice how ES has to ask each shard (either a primary or replica) for the > entire 100k result set. ES has no way of knowing where the results > actually lives - the top N results may all reside on a single shard. As > such, ES has to query each shard for the full result set. In a 5 shard > setup, this means the requesting node will need to merge and sort 500k > results to identify the top 100k. Throw in network latency of actually > transferring 100k documents from each shard to the requesting node (and > then the client), things start to slow down. > > If you do custom sorts or facets, that adds even more memory required > since the appropriate fields need to be fully in memory for sorting. > > On Friday, March 8, 2013 5:54:38 AM UTC-5, Vahid wrote: > > > We already have tested different strategies. At first we had multiple > > queries and result sets(using no filter), as it was not enough fast we > > changed it into one query using filters. We don't use ES paging and each > > node fetch 100k document at once without scrolling. > > At first it was very good and increased our speed 4 time, but after one > > day searching and indexing we got into memory problems and also speed > > decreased to half. Also system is not stable and after some hours I can see > > GC warning messages and ES split brain or no node available exception. > > Any way I'll try again different scenarios. > > > > Best, > > Vahid > > > > On Friday, March 8, 2013 10:07:43 AM UTC+1, Jörg Prante wrote: > > > > > Maybe I missed it. What is your search speed? What do your searches look > > > like, how many hits do you create? I think, in the case you ask ES to > > > create large result sets, you get some extra peaks on the heap, so I > > > assume a search/scroll strategy and smaller result sets could help then. > > > > > > Jörg > > > > > > Am 07.03.13 15:49, schrieb Vahid: > > > > > > > So I'm wondering why the search speed decreased so much?! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:47am UTC](https://discuss.elastic.co/t/es-searching-and-indexing-speed-reduced-after-processing-600milion-records/11035/11 "2017-07-06T02:47:31Z")