How to tune up search performance?

[ted_ye]

hello every experts:
I want to ask for how to tune up search performance?
My cluster:
3 * Master-eligible nodes with 2 cores 4G RAM
3 * coordinating nodes with 2 cores 8G RAM
5 * data nodes with 8cores 32G RAM
elasticsearch version : 7.0.1
there are 0.2 billion logs per day.
And I search all logs of today on kibana, about 0.12 billion items,it takes 20 seconds to return!
!
image|690x182
How can I tune up the search speed? Thanks

[Christian_Dahlqvist]

Have you identified what is limiting performance? Are you saturating CPU? Are you limited by disk utilisation? How large are your indices and shards?

[ted_ye]

I‘ve done nothing to limit the performence.I set each data node jvm RAM with 15G.
I index the logs by date. Everyday has a index , every index has 5 shards and 0 replica,each shard locates on each data node. I set the index sort on timestamp. Is this has any problem?
here is today's index setting:
{
"xxx-xxx-log-2019.09.05" : {
"settings" : {
"index" : {
"lifecycle" : {
"name" : "xxx-log-keep-15days"
},
"refresh_interval" : "5s",
"number_of_shards" : "5",
"provided_name" : "xxx-xxx-log-2019.09.05",
"merge" : {
"scheduler" : {
"max_thread_count" : "1"
}
},
"creation_date" : "1567612807191",
"sort" : {
"field" : "@timestamp"
},
"number_of_replicas" : "0",
"uuid" : "vsewVpzgQsaOfXICv8TwEw",
"version" : {
"created" : "7000199"
}
}
}
}
}
here is the kibana query :

[ted_ye]

I used the search profile to check this problem. I use the kibana to search the last 15 minutes logs, search request is :
{
"version": true,
"size": 500,
"sort": [
{
"@timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"_source": {
"excludes":
},
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "30s",
"time_zone": "Asia/Shanghai",
"min_doc_count": 1
}
}
},
"stored_fields": [
""
],
"script_fields": {},
"docvalue_fields": [
{
"field": "@timestamp",
"format": "date_time"
}
],
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"format": "strict_date_optional_time",
"gte": "2019-09-06T03:21:30.108Z",
"lte": "2019-09-06T03:36:30.108Z"
}
}
}
],
"filter": [
{
"match_all": {}
}
],
"should": [],
"must_not": []
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"": {}
},
"fragment_size": 2147483647
}
}

and the profile is :

the next_doc cost more time, how to make it better?