Elasticsearch replication performance is slow

Elastic Stack Elasticsearch
1

I have an elasticsearch setup that had until now one machine (16core 64GB RAM 4x800GB SSD Ubuntu 16.04 Xenial, containing 1.5TB of log data in 450 indexes, running ES 5.1.1.

Now I added a second, identical server to the cluster. Both are connected via a 10GBit network.

All indexes have 1 shard, and I have configured them to have 1 replica after the second server went online.

Now, replicas are being created, but only slowly. Load on both machines is below 1 and IO rates are at about 2MB/s or less.

I am running the following settings:

{
    "persistent": {
        "cluster": {
            "routing": {
                "allocation": {
                    "node_concurrent_incoming_recoveries": "20",
                    "node_initial_primaries_recoveries": "8",
                    "node_concurrent_outgoing_recoveries": "20"
                }
            }
        },
        "indices": {
            "recovery": {
                "max_bytes_per_sec": "400mb"
            },
            "store": {
                "throttle": {
                    "type": "none"
                }
            }
        }
    },
    "transient": {
        "logger": {
            "org": {
                "elasticsearch": {
                    "indices": "DEBUG"
                }
            }
        }
    }
}

indices.store.throttle.type does not seem to exist anymore in ES 5.

Why is this so slow?
I could not find any other settings or throttles.

Is there anything else to look at?

2

What version exactly?
What OS?

3

ES 5.1.1 on Ubuntu 16.04 Xenial

4

Does anybody have an idea?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.