Search slow when high concurrency?

Elastic Stack Elasticsearch
1

I have a index which need to save about 100 million docs,and the index-type have about 30 fields..

Here's what I do.

template:

{
    "template": "index*",
    "settings": {
        "number_of_shards": 6,
        "number_of_replicas": 1
    },
    "mappings": {
        "type": {
            "_all": {
                "enabled": false
            },
            "dynamic_templates": [
                {
                    "notanalyzed": {
                        "match_mapping_type": "string",
                        "mapping": {
                            "type": "string",
                            "index": "not_analyzed"
                        }
                    }
                }
            ],
            "properties": {
                "field1":{
                     "type": "string"
                },
                "field2":{
                    "type": "string"
                },
                "field3":{
                    "type": "string",
                     "index": "not_analyzed"
                },
                "field4":{
                    "type": "long"
                },
                "field5":{
                    "type": "string",
                     "index": "not_analyzed"
                },
                "field6":{
                    "type": "string",
                     "index": "not_analyzed"
                }
            }
        }
    }
}

dsl:

{
"filtered": {
    "filter": {
        "bool": {
            "should": [
                {
                    "term": {
                        "field3": "xxx"
                    }
                },
                {
                    "range": {
                        "field4": {
                            "lt": "xxx",
                            "gte": "xxx"
                        }
                    }
                }
            ],
            "must": [
                {
                    "term": {
                        "field6": "xxx"
                    }
                }
            ],
            "from": 0,
            "sort": [
                {
                    "field5": "asc"
                }
            ]
        }
    }
}

{"term":{"field6":"?"},field6 about have 1 million different values,and need to be used for query every time, field4 is a time stamp，different query has different value.

server:
3 servers(ssd,98G memory,32 core cpu),each server i deploy 3 nodes,1 master node(10G heap size),2 data nodes(20G heap size).

EDIT:
I find if i use 500 thread to search,and the es response very slow,and many search need be executed exceed 20000 millisecond,and es search.queue sometimes exceed 500,So i want to optimize my cluster or search dsl.

2

Hi @famoss,

500 concurrent search requests against a three node cluster are just too much. It's natural for systems that response times increase, when systems are overloaded. To give you an analogy: Think of 500 customers calling a hotline with 3 agents that answer the phone. Naturally, customers will have to wait. If you're interested in the theory just google for "Little's Law".

So, you should scale horizontally (see the Definitive Guide for specific suggestions). It's also not ideal that you run multiple nodes per machine as they will steal resources from each other (see some tips for running multiple nodes per machine).

Daniel