Query cache gets invalidated

[royal_man]

I am making this query with request_cache=true

{
  "query": {
    "bool": {
      "should": [
        {
          "term": {
            "status": "online"
          }
        },
        {
          "term": {
            "status": "available"
          }
        },
        {
          "term": {
            "status": "admin"
          }
        }
      ]
    }
  }
}

I have refresh interval of 1 minute. Even if I make the same query repeatedly, the cache hit count increases and sometimes miss_count increase. There are no evictions yet. Can someone help what is the issue ?

[spinscale (Alexander Reelsen)]

can you share your full query? Keep in mind that such queries are only cached if size: 0 is specified. See https://www.elastic.co/guide/en/elasticsearch/reference/7.5/shard-request-cache.html

[royal_man]

@spinscale Do I need to specify size:0 explicitly ?

[spinscale (Alexander Reelsen)]

yes, otherwise you will retrieve 10 hits by default

[royal_man]

@spinscale Thanks for your reply. Can you please guide me. How to get the hits also with caching enabled. I have checked that we have to specify size: 0 but I need hits as a primary objective of this query.

Can we change the query a little bit to get hits and also enable caching.

[spinscale (Alexander Reelsen)]

the shard query cache will not help you then.

However there is a second cache named the node query cache, that can speed up queries and does not need to be activated, as it has some heuristics to figure out when it should be used. See https://www.elastic.co/guide/en/elasticsearch/reference/7.5/query-cache.html

Can you explain why you need caching? Maybe there is the possibility to optimize queries and datamodels (for example not requiring an exact hit count might help as well in Elasticsearch 7.x).

[royal_man]

@spinscale I also think shard query cache cannot be used.

I need caching to make queries faster. Since i have a lot of queries which are same so I thought of enabling the cache. I do have refresh interval: 1m

Can you help with any resource to optimize queries and datamodels.
I actually don't need hit count actually. I need hits and want pagination in them.

[DeeeFOX]

@royal_man

{ "query": { "bool": { "should": [ { "term": { "status": "online" } }, { "term": { "status": "available" } }, { "term": { "status": "admin" } } ] } } }

@royal_man to ur use case talking above, only file-system-cache can help.

Just change or append the index setting, with enough free mem in ur data nodes (to our-tsdb-use-case: 31G on-heap + 96G off-heap / per node):

index.store.preload: ["nvd", "dvd", "tim", "doc", "dim"]

or more aggressive

index.store.preload: ["*"]

And reopen ur index, u will see difference!
Wish it can help u!

[royal_man]

Thanks @DeeeFOX I will try the idea and reply here about the performance

[royal_man]

@DeeeFOX
I have tried the filesystem preload and monitored the RAM cache and but there is no change in it.
I have ES data node in k8s node (60Gi RAM). Below is the RAM usage graph. I have actually allocated 50Gi to ES node. I think RAM free includes the 10Gi RAM that is not allocated.
What do you think ?

[DeeeFOX]

Ahhh, sorry for forgetting to remind u, one more thing u should set to enable the preload settings: the index store type, by default is "hybridfs", and to our tsdb-use-case:

{
  "settings": {
    "index": {
      ...
      "store": {
        "type": "mmapfs",
        "preload": [
          "doc",
          "dvd"
        ]
      },
      ...
}