Just a quick update. I changed the setting from 30 seconds to 5 minutes (30000 --> 300000) but still have the same error:
Fatal Error: Courier fetch: Request Timeout after 30000ms
Any clue how to change the setting? I've updated the setting, restarted Kibana, but still the error persists. I'm starting to think that it might be related to Nginx.
I've created a file called/etc/nginx/conf.d/proxy-settings.conf with the following contents:
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 90m;
send_timeout 300;
client_max_body_size 1000m;
https://wiki.ssdt-ohio.org/display/rtd/Adjusting%20nginx-proxy%20Timeout%20Configuration
This seems to have fixed the issue. I'm still a bit concerned since I pushed my system's load average to 7.5 before the query executed successfully.
I guess this is a limitation that should be addressed by adding more / better hardware. I'm kind of running on fumes with 2 CPU / 8 GB RAM for the amount of data that I'm querying. 