hello every experts:
I want to ask for how to tune up search performance?
My cluster:
3 * Master-eligible nodes with 2 cores 4G RAM
3 * coordinating nodes with 2 cores 8G RAM
5 * data nodes with 8cores 32G RAM
elasticsearch version : 7.0.1
there are 0.2 billion logs per day.
And I search all logs of today on kibana, about 0.12 billion items,it takes 20 seconds to return!
!
image|690x182
How can I tune up the search speed? Thanks
Have you identified what is limiting performance? Are you saturating CPU? Are you limited by disk utilisation? How large are your indices and shards?
I‘ve done nothing to limit the performence.I set each data node jvm RAM with 15G.
I index the logs by date. Everyday has a index , every index has 5 shards and 0 replica,each shard locates on each data node. I set the index sort on timestamp. Is this has any problem?
here is today's index setting:
{
"xxx-xxx-log-2019.09.05" : {
"settings" : {
"index" : {
"lifecycle" : {
"name" : "xxx-log-keep-15days"
},
"refresh_interval" : "5s",
"number_of_shards" : "5",
"provided_name" : "xxx-xxx-log-2019.09.05",
"merge" : {
"scheduler" : {
"max_thread_count" : "1"
}
},
"creation_date" : "1567612807191",
"sort" : {
"field" : "@timestamp"
},
"number_of_replicas" : "0",
"uuid" : "vsewVpzgQsaOfXICv8TwEw",
"version" : {
"created" : "7000199"
}
}
}
}
}
here is the kibana query :
I used the search profile to check this problem. I use the kibana to search the last 15 minutes logs, search request is :
{
"version": true,
"size": 500,
"sort": [
{
"@timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"_source": {
"excludes":
},
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "30s",
"time_zone": "Asia/Shanghai",
"min_doc_count": 1
}
}
},
"stored_fields": [
""
],
"script_fields": {},
"docvalue_fields": [
{
"field": "@timestamp",
"format": "date_time"
}
],
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"format": "strict_date_optional_time",
"gte": "2019-09-06T03:21:30.108Z",
"lte": "2019-09-06T03:36:30.108Z"
}
}
}
],
"filter": [
{
"match_all": {}
}
],
"should": [],
"must_not": []
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"": {}
},
"fragment_size": 2147483647
}
}
and the profile is :
the next_doc cost more time, how to make it better?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.