Maybe we should jump back to this for a moment.
Could you please try setting the elasticsearch.requestTimeout to something like 90000 (90 seconds) and see if one of those visualization will load 24 hours of data? I'm thinking that if 12 hours is taking under 30 seconds then 24 hours should take around 60 seconds. We might just inch our way towards you getting everything working.
Another test is to take the request that the "Top hosts creating traffic" visualization is running, and use curl to run that on the Elasticsearch server directly (ssh onto that machine to avoid any network latency, etc.) and see how long it takes. If it's really taking that long, then I think you should start a new post on the Elasticsearch discuss channel because it's not anything that Kibana can do to fix.
time curl http://localhost:9200/packetbeat-*/doc/_search -H "Content-type: application/json" -d '{
"size": 0,
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "1h",
"time_zone": "America/Chicago",
"min_doc_count": 1
},
"aggs": {
"3": {
"terms": {
"field": "source.ip",
"size": 5,
"order": {
"1": "desc"
}
},
"aggs": {
"1": {
"sum": {
"field": "source.stats.net_bytes_total"
}
}
}
}
}
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"*": {}
},
"require_field_match": false,
"fragment_size": 2147483647
},
"_source": {
"excludes": []
},
"stored_fields": [
"*"
],
"script_fields": {},
"docvalue_fields": [
"@timestamp",
"last_time",
"start_time",
"tls.client_certificate.not_after",
"tls.client_certificate.not_before",
"tls.server_certificate.not_after",
"tls.server_certificate.not_before"
],
"query": {
"bool": {
"must": [
{
"match_all": {}
},
{
"query_string": {
"query": "type: flow",
"analyze_wildcard": true,
"default_field": "*"
}
},
{
"range": {
"@timestamp": {
"gte": 1512513175834,
"lte": 1512599575834,
"format": "epoch_millis"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
}
}'
time for the response on my little VM was 95ms, but I have much less data than you. Some of the important details are in the beginning part of the response;
{"took":11,"timed_out":false,"_shards":{"total":10,"successful":10,"skipped":0,"failed":0},"hits":{"total":41741,
