# Elastic query is not fast and response time is high

**URL:** <https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [May 5, 2022, 12:14am UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982 "2022-05-05T00:14:22Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![alpine\_f1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alpine_f1/32/95302_2.png) [@alpine\_f1](https://discuss.elastic.co/u/alpine_f1)\
**Post date:** [May 5, 2022, 12:14am UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982/1 "2022-05-05T00:14:22Z")

</div>

Cluster setup:  
I am running Elasticsearch cluster (2 master, 2 data and 1 voter) as docker containers.  
Kibana setup:  
I also have Kibana configured to connect to the elastic cluster.  
Frontend:  
I have an angular web application that has a search bar using which I can search for accounts etc and show the results in UI.  
Backend:  
My backend is a NodeJS REST api that connects to elastic cluster. I use @elastic/Elasticsearch NPM package to query Elasticsearch indices.  
Issue:  
When I search for an account by typing the account number or account name in the frontend, i make a call to the nodejs REST api in which I am using @elastic/Elasticsearch package to query for the search text or keyword across all the indices (accounts, groups, relationships) and return the results to the UI. My search query’s are lot slower and the response time is around 500 to 800 ms whereas I expect it to be less than 100 ms.

Is there a way to profile or triage this performance issue?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 5, 2022, 12:22am UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982/2 "2022-05-05T00:22:59Z")

</div>

> [@alpine\_f1](#):
>
> My search query’s are lot slower and the response time is around 500 to 800 ms whereas I expect it to be less than 100 ms.

What are those expectations built on?

> [@alpine\_f1](#):
>
> Is there a way to profile or triage this performance issue?

What does the query look like?  
What is the mapping for them?  
What are the node specs?  
What is the output from the `_cluster/stats?pretty&human` API?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 5, 2022, 12:28am UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982/3 "2022-05-05T00:28:12Z")

</div>

You can also capture the query leaving your REST API and go to Kibana - Dev Tools -\> Profiler and analyze the query.

---

<div class="post-metadata">

**Author:** ![alpine\_f1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alpine_f1/32/95302_2.png) [@alpine\_f1](https://discuss.elastic.co/u/alpine_f1)\
**Post date:** [May 5, 2022, 1:03am UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982/4 "2022-05-05T01:03:22Z")

</div>

1. Below is the output of search query:

```auto
{
    "total": {
      "value": 1576,
      "relation": "eq"
    },
    "took": 3048,
    "hits": [],
    "maxScore": null,
    "shards": {
      "total": 7,
      "successful": 7,
      "skipped": 0,
      "failed": 0
    },
    "facets": {
      "relationships": {
        "meta": {},
        "doc_count": 7,
        "relationships": {
          "hits": {
            "total": {
              "value": 7,
              "relation": "eq"
            },
            "max_score": 1,
            "hits": [
              {
                "_index": "relationships",
                "_type": "_doc",
                "_id": "MYDATA-1",
                "_score": 1,
                "_source": {
                  "relId": "MYDATA-1",
                  "relName": "TRENTON dummy data 1"
                },
                "highlight": {
                  "relName": [
                    "<em>TRENTON</em> dummy data 1"
                  ]
                }
              },
              {
                "_index": "relationships",
                "_type": "_doc",
                "_id": "MYDATA-2",
                "_score": 1,
                "_source": {
                  "reId": "MYDATA-2",
                  "relName": "TRENTON dummy data 2"
                },
                "highlight": {
                  "relName": [
                    "<em>TRENTON</em> dummy data 2"
                  ]
                }
              }
            ]
          }
        }
      },
      "accounts": {
        "meta": {},
        "doc_count": 242,
        "accounts": {
          "hits": {
            "total": {
              "value": 242,
              "relation": "eq"
            },
            "max_score": 1,
            "hits": [
              {
                "_index": "accounts",
                "_type": "_doc",
                "_id": "MY-ACC-DATA-1",
                "_score": 1,
                "_source": {
                  "accId": "MYDATA-1",
                  "accName": "TRENTON dummy data 1"
                },
                "highlight": {
                  "accName": [
                    "<em>TRENTON</em> dummy data 1"
                  ]
                }
              },
              {
                "_index": "accounts",
                "_type": "_doc",
                "_id": "MY-ACC-DATA-2",
                "_score": 1,
                "_source": {
                  "accId": "MYDATA-2",
                  "accName": "TRENTON dummy data 2"
                },
                "highlight": {
                  "accName": [
                    "<em>TRENTON</em> dummy data 2"
                  ]
                }
              }
            ]
          }
        }
      },
      "groups": {
        "meta": {},
        "doc_count": 0,
        "groups": {
          "hits": {
            "total": {
              "value": 0,
              "relation": "eq"
            },
            "max_score": null,
            "hits": []
          }
        }
      },
      "index": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "accounts",
            "doc_count": 242
          },
          {
            "key": "relationships",
            "doc_count": 7
          },
          {
            "key": "grops",
            "doc_count": 2
          }
        ]
      }
    },
    "totalMatches": 1576,
    "dynamicQuery": {
      "bool": {
        "must": {
          "query_string": {
            "query": "*trenton*",
            "fields": [
              "*"
            ]
          }
        }
      }
    },
    "totalTime": 3263
}

```

1. Below is the mapping

```auto
accounts: {
  "mappings": {
    "_doc": {
      "properties": {
        "acct_title_txt": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "accId": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "accName": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}
relationships:
{
  "mappings": {
    "_doc": {
      "properties": {
        "relId": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "relName": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

```

1. Below is the output of cluster stats:

```auto
{
  "_nodes": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "cluster_name": "es_cluster",
  "cluster_uuid": "random_hash",
  "timestamp": 1651710673241,
  "status": "green",
  "indices": {
    "count": 80,
    "shards": {
      "total": 160,
      "primaries": 80,
      "replication": 1.0,
      "index": {
        "shards": {
          "min": 2,
          "max": 2,
          "avg": 2.0
        },
        "primaries": {
          "min": 1,
          "max": 1,
          "avg": 1.0
        },
        "replication": {
          "min": 1.0,
          "max": 1.0,
          "avg": 1.0
        }
      }
    },
    "docs": {
      "count": 1971168,
      "deleted": 1197406
    },
    "store": {
      "size": "3.7gb",
      "size_in_bytes": 3987857060,
      "total_data_set_size": "3.7gb",
      "total_data_set_size_in_bytes": 3987857060,
      "reserved": "0b",
      "reserved_in_bytes": 0
    },
    "fielddata": {
      "memory_size": "63.5kb",
      "memory_size_in_bytes": 65032,
      "evictions": 0
    },
    "query_cache": {
      "memory_size": "20.3mb",
      "memory_size_in_bytes": 21310193,
      "total_count": 63035347,
      "hit_count": 2817397,
      "miss_count": 60217950,
      "cache_size": 7297,
      "cache_count": 127506,
      "evictions": 120209
    },
    "completion": {
      "size": "0b",
      "size_in_bytes": 0
    },
    "segments": {
      "count": 454,
      "memory": "8.3mb",
      "memory_in_bytes": 8708600,
      "terms_memory": "5mb",
      "terms_memory_in_bytes": 5334424,
      "stored_fields_memory": "226.8kb",
      "stored_fields_memory_in_bytes": 232304,
      "term_vectors_memory": "0b",
      "term_vectors_memory_in_bytes": 0,
      "norms_memory": "553.6kb",
      "norms_memory_in_bytes": 566912,
      "points_memory": "0b",
      "points_memory_in_bytes": 0,
      "doc_values_memory": "2.4mb",
      "doc_values_memory_in_bytes": 2574960,
      "index_writer_memory": "13.2mb",
      "index_writer_memory_in_bytes": 13876872,
      "version_map_memory": "0b",
      "version_map_memory_in_bytes": 0,
      "fixed_bit_set": "107.8kb",
      "fixed_bit_set_memory_in_bytes": 110400,
      "max_unsafe_auto_id_timestamp": 1651708803088,
      "file_sizes": {}
    },
    "mappings": {
      "field_types": [
        {
          "name": "binary",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "boolean",
          "count": 480,
          "index_count": 36,
          "script_count": 0
        },
        {
          "name": "byte",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "constant_keyword",
          "count": 30,
          "index_count": 10,
          "script_count": 0
        },
        {
          "name": "date",
          "count": 219,
          "index_count": 56,
          "script_count": 0
        },
        {
          "name": "date_nanos",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "date_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "double",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "double_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "float",
          "count": 466,
          "index_count": 32,
          "script_count": 0
        },
        {
          "name": "float_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "geo_point",
          "count": 2,
          "index_count": 2,
          "script_count": 0
        },
        {
          "name": "geo_shape",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "half_float",
          "count": 13,
          "index_count": 2,
          "script_count": 0
        },
        {
          "name": "integer",
          "count": 5,
          "index_count": 2,
          "script_count": 0
        },
        {
          "name": "integer_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "ip",
          "count": 11,
          "index_count": 11,
          "script_count": 0
        },
        {
          "name": "ip_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "keyword",
          "count": 2923,
          "index_count": 65,
          "script_count": 0
        },
        {
          "name": "long",
          "count": 2001,
          "index_count": 46,
          "script_count": 0
        },
        {
          "name": "long_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "nested",
          "count": 8,
          "index_count": 8,
          "script_count": 0
        },
        {
          "name": "object",
          "count": 1283,
          "index_count": 56,
          "script_count": 0
        },
        {
          "name": "shape",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "short",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "text",
          "count": 2439,
          "index_count": 64,
          "script_count": 0
        },
        {
          "name": "version",
          "count": 8,
          "index_count": 8,
          "script_count": 0
        }
      ],
      "runtime_field_types": []
    },
    "analysis": {
      "char_filter_types": [],
      "tokenizer_types": [],
      "filter_types": [],
      "analyzer_types": [],
      "built_in_char_filters": [],
      "built_in_tokenizers": [],
      "built_in_filters": [],
      "built_in_analyzers": [
        {
          "name": "english",
          "count": 1,
          "index_count": 1
        }
      ]
    },
    "versions": [
      {
        "version": "7.15.0",
        "index_count": 32,
        "primary_shard_count": 32,
        "total_primary_size": "868.7mb",
        "total_primary_bytes": 910995300
      },
      {
        "version": "7.16.2",
        "index_count": 48,
        "primary_shard_count": 48,
        "total_primary_size": "990.8mb",
        "total_primary_bytes": 1038932015
      }
    ]
  },
  "nodes": {
    "count": {
      "total": 5,
      "coordinating_only": 0,
      "data": 3,
      "data_cold": 0,
      "data_content": 0,
      "data_frozen": 0,
      "data_hot": 0,
      "data_warm": 0,
      "ingest": 3,
      "master": 3,
      "ml": 0,
      "remote_cluster_client": 5,
      "transform": 0,
      "voting_only": 1
    },
    "versions": [
      "7.16.2"
    ],
    "os": {
      "available_processors": 5,
      "allocated_processors": 5,
      "names": [
        {
          "name": "Linux",
          "count": 5
        }
      ],
      "pretty_names": [
        {
          "pretty_name": "Red Hat Enterprise Linux Server 7.9 (Maipo)",
          "count": 5
        }
      ],
      "architectures": [
        {
          "arch": "amd64",
          "count": 5
        }
      ],
      "mem": {
        "total": "36.6gb",
        "total_in_bytes": 39321600000,
        "free": "14.2gb",
        "free_in_bytes": 15257305088,
        "used": "22.4gb",
        "used_in_bytes": 24064294912,
        "free_percent": 39,
        "used_percent": 61
      }
    },
    "process": {
      "cpu": {
        "percent": 52
      },
      "open_file_descriptors": {
        "min": 448,
        "max": 766,
        "avg": 616
      }
    },
    "jvm": {
      "max_uptime": "14.7d",
      "max_uptime_in_millis": 1276502845,
      "versions": [
        {
          "version": "11.0.9",
          "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
          "vm_version": "11.0.9+7-LTS",
          "vm_vendor": "Oracle Corporation",
          "bundled_jdk": true,
          "using_bundled_jdk": false,
          "count": 5
        }
      ],
      "mem": {
        "heap_used": "7.3gb",
        "heap_used_in_bytes": 7862177080,
        "heap_max": "18.2gb",
        "heap_max_in_bytes": 19617218560
      },
      "threads": 246
    },
    "fs": {
      "total": "124.9gb",
      "total_in_bytes": 134131240960,
      "free": "119.7gb",
      "free_in_bytes": 128550281216,
      "available": "119.7gb",
      "available_in_bytes": 128550281216
    },
    "plugins": [],
    "network_types": {
      "transport_types": {
        "security4": 5
      },
      "http_types": {
        "security4": 5
      }
    },
    "discovery_types": {
      "zen": 5
    },
    "packaging_types": [
      {
        "flavor": "default",
        "type": "tar",
        "count": 5
      }
    ],
    "ingest": {
      "number_of_pipelines": 19,
      "processor_stats": {
        "conditional": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "convert": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "geoip": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "grok": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "gsub": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "remove": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "rename": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "script": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "set": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "set_security_user": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        }
      }
    }
  }
}

```

1. Node specs below:  
All 5 nodes have the same specs which is 7500MB RAM/0.7 core CPU

---

<div class="post-metadata">

**Author:** ![alpine\_f1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alpine_f1/32/95302_2.png) [@alpine\_f1](https://discuss.elastic.co/u/alpine_f1)\
**Post date:** [May 5, 2022, 1:49am UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982/5 "2022-05-05T01:49:59Z")

</div>

I did that. We have aliases loaded with data and we are querying the aliases. The performance on first execution is around 3 seconds and subsequent executions are around 500 ms to 800 ms

---

<div class="post-metadata">

**Author:** ![alpine\_f1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alpine_f1/32/95302_2.png) [@alpine\_f1](https://discuss.elastic.co/u/alpine_f1)\
**Post date:** [May 6, 2022, 11:39am UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982/6 "2022-05-06T11:39:33Z")

</div>

@warkolm/@stephenb - Were you able to review the responses from my cluster?

---

<div class="post-metadata">

**Author:** ![soco](https://avatars.discourse-cdn.com/v4/letter/s/8491ac/32.png) [@soco](https://discuss.elastic.co/u/soco)\
**Post date:** [May 6, 2022, 2:07pm UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982/7 "2022-05-06T14:07:22Z")

</div>

check all node are working could be reply the index any time; can check I/O Disk working , review nginx service if you using that to make reverse proxy

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 6, 2022, 3:03pm UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982/8 "2022-05-06T15:03:57Z")

</div>

> [@alpine\_f1](#):
>
> ```auto
> "query_string": {
> "query": "*trenton*",
> "fields": [
> "*"
> ]
> }
> 
> ```

It looks like you are running a wildcard string query with leading wildcard for a lot of fields, which on a text or keyword type is among the most inefficient queries you can run as a lot of terms need to be checked. Think of it as the equivalent to a full table scan in a relational data base. I suspect you would see much better performance if you mapped your data [as a wildcard type](https://www.elastic.co/guide/en/elasticsearch/reference/8.2/keyword.html#keyword).

> [@alpine\_f1](#):
>
> ```auto
> "total": 160,
> "primaries": 80,
> 
> ```

> [@alpine\_f1](#):
>
> ```auto
> "docs": {
> "count": 1971168,
> "deleted": 1197406
> },
> 
> ```

You also seem to have a lot of small shards which is very inefficient. The amount of data you have would easily fit in a single shard, so I would recommend changing this.

> [@alpine\_f1](#):
>
> 0.7 core CPU

This sounds quite low, especially considering the number of shards you have. I would recommend increasing this and see if it makes a difference.

---

<div class="post-metadata">

**Author:** ![alpine\_f1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alpine_f1/32/95302_2.png) [@alpine\_f1](https://discuss.elastic.co/u/alpine_f1)\
**Post date:** [May 7, 2022, 5:01am UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982/9 "2022-05-07T05:01:48Z")

</div>

Thank you @Christian_Dahlqvist… Let me work on the changes suggested by you and get back on this same thread.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 4, 2022, 5:02am UTC](https://discuss.elastic.co/t/elastic-query-is-not-fast-and-response-time-is-high/303982/10 "2022-06-04T05:02:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.