Strange cachebehaviour

[mfries (Martin Fries)]

Hi,

I have a cachebehaviour which I cannot explain.

Our environment:
4 Nodes
each with 88 cores, 792GB RAM and 20 spinnings disk (RAID10) as datastorage
OS: RHEL 7.6
Docker: docker-1.13.1-94.gitb2f74b2.el7.x86_64 (comes with RHEL)
Elastic-Stack:6.8.0
JVM-HEAP: 31GB
Shards: 1540
Documents: 6,359,903,370
Data: 8,3TB

If I do a:
POST /_cache/clear
as response I get:

{
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  }
}

I would expect 1540 shards. It does not matter how often I do that. It´s always 2 shards.

For example this search from yesterday to 8 days in the past takes about 25 seconds after the initial search.

GET /logstash-*/_search?ignore_unavailable=true&request_cache=true
{
  "version": true,
  "size": 500,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "3h",
        "time_zone": "Europe/Berlin",
        "min_doc_count": 1
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "docker.container.labels.build-date",
      "format": "date_time"
    },
    {
      "field": "mslog2.@timestamp",
      "format": "date_time"
    },
    {
      "field": "syslogtime",
      "format": "date_time"
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1560376800000,
              "lte": 1561067999999,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "filter": [],
      "should": [],
      "must_not": []
    }
  },
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  }
}

Than I do the search with the parameter "profile:true"

GET /logstash-*/_search?ignore_unavailable=true&request_cache=true
{
  "version": true,
  "profile": "true",
  "size": 500,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "3h",
        "time_zone": "Europe/Berlin",
        "min_doc_count": 1
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "docker.container.labels.build-date",
      "format": "date_time"
    },
    {
      "field": "mslog2.@timestamp",
      "format": "date_time"
    },
    {
      "field": "syslogtime",
      "format": "date_time"
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1560376800000,
              "lte": 1561067999999,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "filter": [],
      "should": [],
      "must_not": []
    }
  },
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  }
}

I´m not interested in the result. But that seems to fix the cache.
After that I do a
POST /_cache/clear
and as response I get:

{
  "_shards" : {
    "total" : 1540,
    "successful" : 1540,
    "failed" : 0
  }
}

The search without the "profile:true" than takes about 1 second. After the first initial search.
It seems that the cache is working as expected for about 24 hours or something like that. At this time I could not exactly say.
Has anyone an idea what´s going on here?

thanks
Martin