# ES not returning results when doing multi-thread msearch **URL:** \ **Category:** Elasticsearch\ **Created:** [July 13, 2018, 1:58am UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855 "2018-07-13T01:58:07Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![paulshadepanda](https://avatars.discourse-cdn.com/v4/letter/p/bbe5ce/32.png) [@paulshadepanda](https://discuss.elastic.co/u/paulshadepanda)\ **Post date:** [July 13, 2018, 1:58am UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/1 "2018-07-13T01:58:08Z")
A little background. Below is probably not what ElasticSearch is designed for but this is the most efficient way I can think of for achieving what I want to achieve. Setup and Goal: I have an index (ES 1.7), 800k documents, and \<500M in size (one doc is pretty small). I have \> 1 million queries that I need to run against this index and only returning the top 25 documents each time. I want to be able to achieve this below 1 hour. Current Failed Attempt (I have tried many other ways but all failed, all end up with this same problem): 1. I am using Python requests package; 2. The queries are partitioned into 60 partitions, each partition around 20000 queries; 3. I created 15 identical indexes (2 shards, 0 replica); 4. I loop through the partitions doing the following: 4a. I am using the `msearch` end point; 4b. I am using multiprocessing with 15 processes (one process using one of the 15 identical indexes); 4c. Each process, I do 10 queries at a time through `msearch`. 5. One partition (at least for the first 3) actually only takes around 20 seconds to finish. 6. When I loop to the 4th partition, problem arises. I am getting empty responses (response.text is empty). 7. I waited for a little to run the 4th partition, it finished successfully. 8. The 5th partition failed again. 9. I waited for a longer time. I can run 5th, 6th and 7th but the 8th failed. My suspicion is that: there might be some caching going on. When I run many queries at the short time period, the caching place gets filled; when I stop querying, the cache got released more and more as quiet time gets longer and longer. I am no engineer just a user so I don't really know what is going on under the hood. Can someone point to me what the problem is? Is there anything I can do to stop that "caching"? Thanks in advance, Wei ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2018, 7:14am UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/2 "2018-07-13T07:14:30Z")
> [@paulshadepanda](#): > > I have an index (ES 1.7) Why are you using such an old version? > [@paulshadepanda](#): > > 800k documents, and \<500M in size (one doc is pretty small) Given the size a single primary shard might be better. As you do not seem to be writing to it, you may also benefit from force merging it down to a single segment. > [@paulshadepanda](#): > > I created 15 identical indexes (2 shards, 0 replica); Why have you done this? What is the specification of your cluster? Do you have monitoring installed so you can see what is limiting performance? ---
**Author:** ![paulshadepanda](https://avatars.discourse-cdn.com/v4/letter/p/bbe5ce/32.png) [@paulshadepanda](https://discuss.elastic.co/u/paulshadepanda)\ **Post date:** [July 13, 2018, 4:37pm UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/3 "2018-07-13T16:37:23Z")
> [@Christian\_Dahlqvist](#): > > > [@paulshadepanda](#): > > > > I have an index (ES 1.7) > > Why are you using such an old version? I don't have control over this. > [@Christian\_Dahlqvist](#): > > > [@paulshadepanda](#): > > > > I created 15 identical indexes (2 shards, 0 replica); > > Why have you done this? I once did it on a single index and the operations made the index to go red. > [@Christian\_Dahlqvist](#): > > What is the specification of your cluster? Do you have monitoring installed so you can see what is limiting performance? Below are the stats from running `_cluster/stats`: { timestamp: 1531499575241, cluster\_name: "dev-esapp2-17", status: "green", indices: { count: 56, shards: { total: 363, primaries: 193, replication: 0.8808290155440415, index: { shards: { min: 1, max: 15, avg: 6.482142857142857, }, primaries: { min: 1, max: 5, avg: 3.4464285714285716, }, replication: { min: 0, max: 2, avg: 0.75, }, }, }, docs: { count: 87904331, deleted: 16087362, }, store: { size\_in\_bytes: 444425411554, throttle\_time\_in\_millis: 1773336, }, fielddata: { memory\_size\_in\_bytes: 228118236, evictions: 0, }, filter\_cache: { memory\_size\_in\_bytes: 2601705780, evictions: 0, }, id\_cache: { memory\_size\_in\_bytes: 0 }, completion: { size\_in\_bytes: 0 }, segments: { count: 6315, memory\_in\_bytes: 2415717542, index\_writer\_memory\_in\_bytes: 0, index\_writer\_max\_memory\_in\_bytes: 6912139264, version\_map\_memory\_in\_bytes: 0, fixed\_bit\_set\_memory\_in\_bytes: 23999296, }, percolate: { total: 0, time\_in\_millis: 0, current: 0, memory\_size\_in\_bytes: -1, memory\_size: "-1b", queries: 0, }, }, nodes: { count: { total: 15, master\_only: 3, data\_only: 9, master\_data: 0, client: 0, }, versions: [ "1.7.5" ], os: { available\_processors: 108, mem: { total\_in\_bytes: 731218132992 }, cpu: [ { vendor: "Intel", model: "Xeon", mhz: 2300, total\_cores: 8, total\_sockets: 1, cores\_per\_socket: 8, cache\_size\_in\_bytes: 46080, count: 15, } ], }, process: { cpu: { percent: 143 }, open\_file\_descriptors: { min: 547, max: 1798, avg: 1289, }, }, jvm: { max\_uptime\_in\_millis: 2056273349, versions: [ { version: "1.8.0\_151", vm\_name: "OpenJDK 64-Bit Server VM", vm\_version: "25.151-b12", vm\_vendor: "Oracle Corporation", count: 8, }, { version: "1.8.0\_102", vm\_name: "OpenJDK 64-Bit Server VM", vm\_version: "25.102-b14", vm\_vendor: "Oracle Corporation", count: 7, }, ], mem: { heap\_used\_in\_bytes: 160690785248, heap\_max\_in\_bytes: 350172217344, }, threads: 1426, }, fs: { total\_in\_bytes: 4779726888960, free\_in\_bytes: 3695000637440, available\_in\_bytes: 3451921522688, }, plugins: [ { name: "head", version: "NA", description: "No description found.", url: "/\_plugin/head/", jvm: false, site: true, }, { name: "cloud-aws", version: "2.7.1", description: "Cloud AWS Plugin", jvm: true, site: false, }, ], }, } What kind of monitoring are you referring to? Which tool should I use? Which fields should I pay attention to? Thank you so much, Wei ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2018, 4:48pm UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/4 "2018-07-13T16:48:52Z")
I do not see how having 15 identical indices will help performance. If you just had a single one it would most certainly be cached. What did you do that made it go red? I would recommend keeping just one of the identical indices (possibly with a single primary shard) and increase the number of replicas so all data nodes in the cluster have a copy, e.g. 8 if you have 9 data nodes. Before you increase the number of replicas, run [optimize](https://www.elastic.co/guide/en/elasticsearch/reference/1.7/indices-optimize.html) on it with `max_num_segments` set to `1`. Do you have any non-default settings in your config? ---
**Author:** ![paulshadepanda](https://avatars.discourse-cdn.com/v4/letter/p/bbe5ce/32.png) [@paulshadepanda](https://discuss.elastic.co/u/paulshadepanda)\ **Post date:** [July 13, 2018, 5:02pm UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/5 "2018-07-13T17:02:02Z")
I did not know what exactly caused the index to go red since the cluster at the time was not stable as I later found out. But what I did was using `msearch` with 20000 queries at the same time and I did this 3 times. Not sure if this is the cause. I've tried it with more replicas (6) but with 2 primary shards (we have 15 nodes). Let me try with 1 primary shards and 13 replica? For the indexes, other than max\_result\_winodw is set to 3000000, everything is default. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2018, 5:06pm UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/6 "2018-07-13T17:06:58Z")
The stats you provided indicate that you have 9 data nodes, which is why I suggested 8 replicas. > [@paulshadepanda](#): > > But what I did was using `msearch` with 20000 queries at the same time That seems excessive. It is better to send a larger number of smaller batches (100?) rather than a few large ones as it uses up less resources. Make sure you distribute requests across all data nodes in the cluster. ---
**Author:** ![paulshadepanda](https://avatars.discourse-cdn.com/v4/letter/p/bbe5ce/32.png) [@paulshadepanda](https://discuss.elastic.co/u/paulshadepanda)\ **Post date:** [July 13, 2018, 5:10pm UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/7 "2018-07-13T17:10:28Z")
> [@Christian\_Dahlqvist](#): > > Make sure you distribute requests across all data nodes in the cluster. How do I achieve this? Does ES automatically do that or I have to manually direct requests to specific nodes? Thanks. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2018, 5:14pm UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/8 "2018-07-13T17:14:52Z")
How are you querying the data? Using one of the language clients? ---
**Author:** ![paulshadepanda](https://avatars.discourse-cdn.com/v4/letter/p/bbe5ce/32.png) [@paulshadepanda](https://discuss.elastic.co/u/paulshadepanda)\ **Post date:** [July 13, 2018, 5:21pm UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/9 "2018-07-13T17:21:20Z")
I am using Python requests package uri = '[http://xxx:9200/index1/\_msearch](http://xxx:9200/index1/_msearch)' response = requests.get(uri, data=body) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2018, 5:22pm UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/10 "2018-07-13T17:22:25Z")
Then you can provide the address of multiple nodes to it and it will distribute requests across these. ---
**Author:** ![paulshadepanda](https://avatars.discourse-cdn.com/v4/letter/p/bbe5ce/32.png) [@paulshadepanda](https://discuss.elastic.co/u/paulshadepanda)\ **Post date:** [July 13, 2018, 5:29pm UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/11 "2018-07-13T17:29:54Z")
I only know one url that points to these multiple nodes. This one url is sufficient, right? It will just distribute requests across them automatically? Sorry if the question seems dumb. Thanks, Wei ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 10, 2018, 5:29pm UTC](https://discuss.elastic.co/t/es-not-returning-results-when-doing-multi-thread-msearch/139855/12 "2018-08-10T17:29:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.