Preloading all documents to filesystem cache

[9guar1 (Сергей Короленко)]

Hi community,

Is there any possibility to preload all documents to the filesystem cache using 'index.store.preload' to increase search perfomance of the fetch phase?

Search query is below:

POST /structure/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&scroll=60000ms&search_type=dfs_query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true
{
    "from": 0,
    "size": 10000,
    "version": true,
    "explain": false,
    "profile": true,
    "_source": {
        "include": ["externalId"],
        "exclude":[]
    },
    "query": {
        "bool": {
            "must": [
                {
                    "term": {
                        "active": {
                            "value": true,
                            "boost": 1.0
                        }
                    }
                },
                {
                    "bool": {
                        "should": [
                            {
                                "terms": {
                                    "path.parent": [
                                        "022D329A-25EA-E711-80FD-001DD8B71DB5"
                                    ],
                                    "boost": 1.0
                                }
                            },
                            {
                                "terms": {
                                    "path.parent": [
                                        "4060741400016833756"
                                    ],
                                    "boost": 1.0
                                }
                            },
                            {
                                "bool": {
                                    "must": [
                                        {
                                            "bool": {
                                                "must": [
                                                    {
                                                        "terms": {
                                                            "data.engagementModel": [
                                                                "Managed Capacity"
                                                            ],
                                                            "boost": 1.0
                                                        }
                                                    },
                                                    {
                                                        "term": {
                                                            "type": {
                                                                "value": "STREAM",
                                                                "boost": 1.0
                                                            }
                                                        }
                                                    }
                                                ],
                                                "adjust_pure_negative": true,
                                                "boost": 1.0
                                            }
                                        },
                                        {
                                            "terms": {
                                                "path.parent": [
                                                    "8503000000000008968"
                                                ],
                                                "boost": 1.0
                                            }
                                        }
                                    ],
                                    "adjust_pure_negative": true,
                                    "boost": 1.0
                                }
                            },
                            {
                                "terms": {
                                    "path.parent": [
                                        "335FC821-8BD7-E411-80CF-001DD8B71C26",
                                        "6743FA4B-DD01-E811-80FF-001DD8B71DB5",
                                        "C9818B08-206E-EC11-AD3B-00155D00AC2C",
                                        "F2FFA6FA-1F6E-EC11-AD3B-00155D00AC2C",
                                        "32FB7711-860C-EC11-AD33-00155D00AC2C",
                                        "B7534C9C-C643-EC11-AD38-00155D00AC2C",
                                        "34D3E192-A476-E711-80F0-001DD8B71DB5",
                                        "F821EB2A-8ED7-E411-80CF-001DD8B71C26",
                                        "25151EAF-757A-EB11-AD28-00155D00AC29",
                                        "D252FA57-E3F5-EB11-AD32-00155D00AC2C",
                                        "1071F214-16DF-EB11-AD31-00155D00AC2C",
                                        "B3D6D3B5-E298-EA11-AD10-00155D00A803",
                                        "34C3BD1B-8680-E911-811E-001DD8B71DB5",
                                        "3671F1B1-A476-E711-80F0-001DD8B71DB5",
                                        "B61057CC-777A-EB11-AD28-00155D00AC29",
                                        "00E035D3-3263-EC11-AD3C-00155D01722B",
                                        "72A64A3F-767A-EB11-AD28-00155D00AC29",
                                        "370FDA97-DD01-E811-80FF-001DD8B71DB5",
                                        "8E7D5F86-767A-EB11-AD28-00155D00AC29",
                                        "B9116001-9AA2-EB11-AD29-00155D017224",
                                        "123F0385-0CC5-EB11-AD2E-00155D01722B",
                                        "72A14D78-99A2-EB11-AD29-00155D017224",
                                        "06E614E3-98A2-EB11-AD29-00155D017224",
                                        "AA9090EE-97A2-EB11-AD29-00155D017224",
                                        "321758B4-1B53-EA11-AD0C-00155D018201",
                                        "048FC792-AA5A-E911-811C-001DD8B71DB5",
                                        "510EF48E-AB5A-E911-811C-001DD8B71DB5",
                                        "788872ED-AB5A-E911-811C-001DD8B71DB5"
                                    ],
                                    "boost": 1.0
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1.0
        }
    }
}

Running result:

1. The first run took 907 ms and fetch phase ~ 700 ms.
   
   

   

2. The second run took 249 ms and fetch phase ~ 212 ms.
   
   

   

As see the second run is mush faster then the first. Is the reason in the filesystem cache and loaded documents to memory or any other reasons?

[warkolm (Mark Walkom)]

Welcome to our community!

It could be a bit of both, yes. And yes, you can use that setting.

[9guar1 (Сергей Короленко)]

Hi,

I tried to use this setting with ["nvd", "dvd"] and ["*"], but the first execution of the query still was the same (~ 800-900 ms), but i expected it was faster. I set this setting on created index with full data (used api to close index, then set index.store.preload, then opened index, and then stopped and rerun docker container). May be I don't understand something. Could you please give me advice what i can try to do more or share with me additional documentation about this setting?