I put about 1,000,000,000 data into a index of elasticsearch. The index has 10 shards. Each data has a field that is a time and the time is not analyzed. I range query data by the time. I use 100 threads to search it. The speed is very slow. Every thread cost about 10s. How can I deal with the problem?
Can you provide a search example?
How large are the shards? What is the specification of your cluster/machine? What is the query latency if you only use a single thread for querying? What is the query latency if you use 10 threads for querying?
Every shard had about 80G data.
The cluster has two nodes. Every node is:
- 3T SSD
- 24 CPU,E5 2620 v2
- 128G memory, I offered 64G for elasticsearch
A single thread cost about 1s~2s.
The example:
RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("time").from(1400000000).to(1500000000);
The time is absolute number of seconds. And the size I set was 1000.