Here's a summary of my environment (ommiting some private info):
User -> Apache (port:443) -> Kibana (port:5601) -> Elasticsearch (port:9200)
Elasticsearch:
{
"status" : 200,
"version" : {
"number" : "1.4.4",
"build_hash" : "c88f77ffc81301dfa9dfd81ca2232f09588bd512",
"build_timestamp" : "2015-02-19T13:05:36Z",
"build_snapshot" : false,
"lucene_version" : "4.10.3"
},
"tagline" : "You Know, for Search"
}
{
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 2703,
"active_shards" : 5406,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
Elasticsearch is ingesting ~120GB of logs/day.
Kibana:
Version 4.1.2
Build 7562
Commit SHA 34af16e
Apache:
Server version: Apache/2.4.6 (Red Hat)
Server built: Dec 4 2014 03:30:10
In this setup, everything works fine, except when i try to select a time range longer than 24h on any dashboard. When i do so, selecting for exemple "this week" as the time range, i get the following error on kibana after ~ 2 minutes:
Error: Bad Gateway
at respond (https://www-logs.prevnet/index.js?_b=7562:85289:15)
at checkRespForFailure (https://www-logs.prevnet/index.js?_b=7562:85257:7)
at https://www-logs.prevnet/index.js?_b=7562:83895:7
at wrappedErrback (https://www-logs.prevnet/index.js?_b=7562:20902:78)
at wrappedErrback (https://www-logs.prevnet/index.js?_b=7562:20902:78)
at wrappedErrback (https://www-logs.prevnet/index.js?_b=7562:20902:78)
at https://www-logs.prevnet/index.js?_b=7562:21035:76
at Scope.$eval (https://www-logs.prevnet/index.js?_b=7562:22022:28)
at Scope.$digest (https://www-logs.prevnet/index.js?_b=7562:21834:31)
at Scope.$apply (https://www-logs.prevnet/index.js?_b=7562:22126:24)
On the apache log, i get the following message:
[Sat Nov 14 16:15:47.212271 2015] [proxy_http:error] [pid 1661:tid 140115036788480] (20014)Internal error: [client 10.10.10.10:38893] AH01102: error reading status line from remote server 127.0.0.1:5601, referer: https://www-logs.prevnet/
[Sat Nov 14 16:15:47.212385 2015] [proxy:error] [pid 1661:tid 140115036788480] [client 10.0.169.82:38893] AH00898: Error reading from remote server returned by /elasticsearch/_msearch, referer: https://www-logs.prevnet/
No errors are logged on the Elasticsearch servers.
Apache config:
<VirtualHost *:443>
ServerName www-logs
ServerAlias www-logs.prevnet
DocumentRoot /var/www/www-logs
ServerPath /var/www/www-logs
TimeOut 1200
ProxyPreserveHost On
ProxyRequests On
ProxyVia On
<Location "/">
LogLevel warn
ProxyPass http://127.0.0.1:5601/ connectiontimeout=1200 timeout=1200 Keepalive=On
ProxyPassReverse http://127.0.0.1:5601/
SetEnv proxy-initial-not-pooled
</Location>
LogLevel warn
CustomLog /var/log/httpd/www-logs-ssl_access.log combined
ErrorLog /var/log/httpd/www-logs-ssl_error.log
SSLEngine On
SSLProtocol all -SSLv2 -SSLv3
SSLOptions +StdEnvVars +ExportCertData
SSLProxyEngine On
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL:!MD5
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
ProxyPreserveHost On
SSLVerifyDepth 10
SSLCertificateFile /path_to_my_certs/my_certfile.crt
SSLCertificateKeyFile /path_to_my_certs/my_keyfile.key
SSLCertificateChainFile /path_to_my_certs/my_chainfile.crt
SSLCACertificateFile /path_to_my_certs/my_chainfile.crt
</VirtualHost>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 5
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
BrowserMatch "MSIE [2-5]" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
Kibana config:
port:
host: ""
elasticsearch_url: "http://my_elasticsearch_server:9200/"
elasticsearch_preserve_host: false
kibana_index: ".kibana"
default_app_id: "dashboard/Default"
request_timeout: 300000
shard_timeout: 0
verify_ssl: true
pid_file: /var/run/kibana.pid
bundled_plugin_ids:
- plugins/dashboard/index
- plugins/discover/index
- plugins/doc/index
- plugins/kibana/index
- plugins/markdown_vis/index
- plugins/metric_vis/index
- plugins/settings/index
- plugins/table_vis/index
- plugins/vis_types/index
- plugins/visualize/index
As you can see on the above apache config, i've already tryed to increase the apache timeout without success.
Any ideas?
Thanks!