Really slow speed search ES 7.8

Elastic Stack Elasticsearch
1

hi guys, i just starting with ES 7.8 and i'm getting really slow search perfoming, i'm loading data from mysql to ES with this structure:

https://paste.ofcode.org/pmYEUVmWGKGvwxr2FEJZvz

and i'm trying to read it from php with this code:

https://paste.ofcode.org/33JmCWw3zB3wbPJ8iCNkBbm

i just run it for 40 minutes and didn't end :pensive:

2

Can you run your query in Kibana (or cURL) to see how it performs, or limit it more to get any data? What's going on with the cluster during the search, i.e. CPU, any GC, etc.? Turn monitoring on and you can also profile a query in Kibana.

3

@Shryx_Cetus
How many matches are you expecting?

4

Why are you using a size of 10? What happens if you increase this to 1000 or 10000?

5

hi Christian, thank for you answer, i change the size to 10000 and it's more faster, but it's still slow, i'm getting something like a read of 30000 documents every 0,3 seconds

6

Hi vinayak, thank for your answer, i'm expecting to match a maximun of 15 atributes and 1 range in this case a date.

7

How many results are you expecting to match? What is the specification of your cluster? What type of disk/storage do you have?

8

hi steve, i test the query in kibana and i get a similar perform, for a other test i try to get a smaller size data and it took 0.3 in around of 30000 documents, i finally i test it with a biggest size of data and i'm get this monitoring

Captura de pantalla de 2020-07-19 10-15-45
Captura de pantalla de 2020-07-19 10-15-45702×542 66.2 KB

as a additional information, i'm trying to do something like a sum with a group by, for now, i'm just searching the data with the matching filtres and then sum and
"group" with php, could be a best idea try an aggregation?

9

hi christian, i'm expeting to match all, because it's posible to get a report which except all the data(4.8 mill of documents), for know, i'm testing it in a personal computer it have a 16gb ram, i7 4700m with a 256 gb ssd, for now i didn't testing in a server because i didn't have it :sweat_smile:

10

Well, RAM/Disk okay so seems CPU bound, suggest you profile the query in Kibana (I've not done that much, so can't help much) to see how you might improve it.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.