A little background. Below is probably not what ElasticSearch is designed for but this is the most efficient way I can think of for achieving what I want to achieve. Setup and Goal: I have an index (ES 1.7), 800k documents, and <500M in size (one doc is pretty small). I have > 1 million queries tha…

[image] paulshadepanda: I have an index (ES 1.7) Why are you using such an old version? [image] paulshadepanda: 800k documents, and <500M in size (one doc is pretty small) Given the size a single primary shard might be better. As you do not seem to be writing to it, you may also benef…

[image] Christian_Dahlqvist: [image] paulshadepanda: I have an index (ES 1.7) Why are you using such an old version? I don't have control over this. [image] Christian_Dahlqvist: [image] paulshadepanda: I created 15 identical indexes (2 shards, 0 replica); Why have you done …

I do not see how having 15 identical indices will help performance. If you just had a single one it would most certainly be cached. What did you do that made it go red? I would recommend keeping just one of the identical indices (possibly with a single primary shard) and increase the number of re…

I did not know what exactly caused the index to go red since the cluster at the time was not stable as I later found out. But what I did was using msearch with 20000 queries at the same time and I did this 3 times. Not sure if this is the cause. I've tried it with more replicas (6) but with 2 prima…

The stats you provided indicate that you have 9 data nodes, which is why I suggested 8 replicas. [image] paulshadepanda: But what I did was using msearch with 20000 queries at the same time That seems excessive. It is better to send a larger number of smaller batches (100?) rather than a few…

[image] Christian_Dahlqvist: Make sure you distribute requests across all data nodes in the cluster. How do I achieve this? Does ES automatically do that or I have to manually direct requests to specific nodes? Thanks.

How are you querying the data? Using one of the language clients?

I am using Python requests package uri = ' http://xxx:9200/index1/_msearch ' response = requests.get(uri, data=body)

Then you can provide the address of multiple nodes to it and it will distribute requests across these.

I only know one url that points to these multiple nodes. This one url is sufficient, right? It will just distribute requests across them automatically? Sorry if the question seems dumb. Thanks, Wei

ES not returning results when doing multi-thread msearch

Elastic Stack Elasticsearch

system (system) Closed August 10, 2018, 5:29pm 12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ElasticSearch msearch Elasticsearch	1	766	July 5, 2017
Msearch performance decreases when increasing the number of shards Elasticsearch	8	1533	October 1, 2018
Tune Elasticsearch to process thousands of simultaneous searches Elasticsearch	16	1444	March 7, 2023
Elasticsearch multi threading problems and How to improve elasticsearch performance Elasticsearch	15	8646	September 28, 2018
Elasticsearch _msearch api useless? Elasticsearch	1	732	July 10, 2017

ES not returning results when doing multi-thread msearch

Related topics