Queries get slow while indexing documents

Elastic Stack Elasticsearch
1

I have an index with approximately 2.3M documents.

It usually returns results of query within 1~2 seconds, but it gets very slow (sometimes as long as 20~30s) while I'm inserting data into the index.

I insert data in RDB into the index in a batch process using Bulk API (about 100 documents per second).

Why does this happen?
I could not find any documentations about the behavior.

2

What kind of query are you running? Could you share an example?
What is the response? Could you share are least the first 20 lines?

Also what kind of hardware do you have? Heap size, SSD vs HDD...

3

Counting all documents ( like _count {"query": "match_all": {}} ).
Other search queries also sometimes get slow.

I cannot share the response, but response size is small.

I'm using m5.xlarge.elasticsearch * 2 (4 vCPU, 16 GB RAM, 512 GB SSD) for data nodes.

4

Why? There's nothing secret in the first lines.

1 Like
5

BTW a match_all taking normally some seconds is a way too slow.

What is the output of:

GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v

If some outputs are too big, please share them on gist.github.com and link them here.

6

Indexing can be both CPU and dusk I/O intensive. I do not know what monitoring you have access to but it would be good to try to identify if CPU or dusk I/O is limiting performance. If you are using gp2 EBS it gets IOPS proportional to size and since large scroll queries results in a lot of disk I/O so that is probably what I would start with.

1 Like
7

Sorry, search response is like this (just one of examples):

{
    "took": 6554,
    "timed_out": false,
    "_shards": {
        "total": 2,
        "successful": 2,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 10000,
            "relation": "gte"
        },
        "max_score": 6.086506,
        "hits": [...]
    }
}

count response is:

{
    "count": 2300000,
    "_shards": {
        "total": 2,
        "successful": 2,
        "skipped": 0,
        "failed": 0
    }
}
8

I created gist for this.

9

You may also look at making indexing more efficient, e.g. by increasing the refresh interval of the index if you have not already done so. You might even disable it during the bulk load and just enable it afterwards.

2 Likes
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.