Hi i am getting this error (Discover throws error: "An error occured with your request. Reset your inputs and try again") in kibana when i try to do a search of time length 7 days or longer.
I modified the request_timeout setting in kibana.yml and restarted kibana
Still throwing the same error.
There is no errors in the elasticsearch client node that kibana is connected to.
I checked a bunch of the data nodes and i see GC happening during the time of the long search. When i run a 7 day query, is elasticsearch trying to load all 7TB of indices data into memory and that is why the kibana query is failing?
I have 400GB of heap memory and 37TB of indices data. Are there any tweaks that i can do to make large queries work or am i limited to querying a smaller subset of data.