Searchable snapshot

Hello,

I’m conducting performance tests on Searchable Snapshots and would like to discuss my approach to see if it is correct.

Currently, I’ve created a snapshot repository on NAS to generate a 100GB partial shard for performance testing.

As I understand, in the frozen tier, when searching the entire data, only the recently searched parts are kept in the local cache and the search results are returned using this cache. The size of this cache is controlled by the xpack.searchable.snapshot.shared_cache.size setting. I also understand that it is recommended to set xpack.searchable.snapshot.shared_cache.size to 5-10% of the total disk volume. So, if the total volume is 100GB, the recommended xpack.searchable.snapshot.shared_cache.size would be up to 10GB.

I am currently conducting tests by sequentially reducing the xpack.searchable.snapshot.shared_cache.size (to values like 1GB, 500MB, etc.) to examine the degree to which searchability or search speed declines based on the (cache:data_size) ratio.

My questions are:

  1. Are there any aspects of my current testing approach that need to be corrected (e.g., should I adjust request_cache settings)?

  2. From the tests conducted so far:

  • I understand that increasing the cache size should result in faster response times. However, my tests show that when I apply a specific filter (term: {"field": {"value": "{{value}}"}}), increasing the cache size actually slows down the response time. On the other hand, when no filter is applied (term: {"field": {"value": "{{value}}"}}), increasing the cache size does improve response time. I would like to know if my understanding is incorrect.

Example test configurations:

xpack.searchable.snapshot.shared_cache.size : 10GB

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_10gb/_search?request_cache=false

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_10gb/_search

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_1gb/_search?request_cache=false

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_1gb/_search

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

xpack.searchable.snapshot.shared_cache.size : 5GB

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_10gb/_search?request_cache=false

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_10gb/_search

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_1gb/_search?request_cache=false

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_1gb/_search

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

xpack.searchable.snapshot.shared_cache.size : 1GB

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_10gb/_search?request_cache=false

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_10gb/_search

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_1gb/_search?request_cache=false

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search?request_cache=false -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_1gb/_search

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"size":10000}'

curl -k -u {{elastic_id}}:{{passward}} https://host_ip:9200/index_name_100gb/_search -H "Content-Type: application/json" -d '{"query":{"term":{"field":{"value":"{{value}}"}}}}'

.... continue with down cache

ping