Elasticsearch occasionally slow query problem

[binzhang]

Basic configurations:
cluster: two nodes as master eligible node and data node
es version: 7.2.0
jvm heap size: 31GB per node
node memory: 256GB per node
primary/replica shards: 1/1
disk: 2TB+ per node
deploy mode: docker
host system: debian 8
deploy configure:

•     docker run -d --name esnode01 \*
  

•   	-v /home/workspace/elasticsearch/data:/usr/share/elasticsearch/data \*
  

•   	--ulimit memlock=-1:-1 \*
  

•   	--ulimit nofile=65535:65535 \*
  

•   	-p 9200:9200 \*
  

•   	-p 9300:9300 \*
  

•   	-e node.name=esnode01 \*
  

•   	-e node.data=true \*
  

•   	-e node.master=true \*
  

•   	-e network.publish_host=$ip01 \*
  

•   	-e discovery.seed_hosts=$ip02:9300 \*
  

•   	-e cluster.initial_master_nodes=esnode01,esnode02 \*
  

•   	-e cluster.name=es-docker-cluster \*
  

•   	-e bootstrap.memory_lock=true \*
  

•   	-e "ES_JAVA_OPTS=-Xms31g -Xmx31g" \*
  

•   	-e http.cors.enabled=true \*
  

•           -e http.cors.allow-origin=* \*
  

•   	--restart=always \*
  

•   	docker.elastic.co/elasticsearch/elasticsearch:7.2.0*
  

  index mappings:
  {
  "mapping": {
  "properties": {
  "question": {
  "type": "text"
  },
  "update_time": {
  "type": "date"
  }
  }
  }
  }
  index docs: 11k
  ps:
  1. column question are chinese words
  2. enabled search slow log
     "index.search.slowlog.level": "debug",
     "index.search.slowlog.threshold.fetch.debug": "5ms",
     "index.search.slowlog.threshold.query.debug": "1ms",

Query pharse：
POST /index_name/_search
{
"from": 0,
"size": 50,
"query": {
"match": {
"question": text
}
}
}

Problem:
Most queries cost time < 20 ms, Occasionally, 'took' field equals from 100ms, 200ms, up to 900+ms..

Analysis:
kibana monitoring shows no high indexing rate, no long gc time, no high system load, no two many segments, even no high search and query lantency;
search_slow_log shows no high took time querys, profile api shows very small query time_in_nano

Any advices are appreciated

[binzhang]

Supplementation:
no high parallel queries!!!

es monitoring query and fetch time in last 6 hours(unit: ms):

es query service took time in last 6 hours(unit: second):