Hello,
Maybe someone can help me. My setup:
AWS Servers using rsyslog (UTC time) > Physical server in datacenter
central syslog-ng server (CST).
Logstash shipper is running on the central syslog-ng box (CST). It grabs
the events coming in, mangles them, throws them into redis. Logstash
indexer on another box grabs them out of redis, shoves them in
elasticsearch.
Everything works as expected for months now, the only problem I have is
that the display in Kibana doesn't show the log events for 5 hours because
of the Logstash shipper being CST (5 hours behind). Any idea on how to get
it to display immediately? Logs display immediately if I send to the
central log server from a server that is CST as well. Here is a sample from
an AWS box (UTC) that is picked up by the central log server (CST)
Is there any way to get Kibana to show the events as they come in
correctly? We have lots of physical machines in our datacenters and they
are all set to CST, but all of our AWS instances are set to UTC. As of
right now, we don't want to change the central syslog server's timezone to
UTC since it still resides in one of our data centers.
Any ideas? Is this something we should try to fix at the Logstash config or
is this a display fix for Kibana?
Here is a sample from an AWS box (UTC) that is picked up by the central log server (CST) - Displays 5 hours later/incorrectly
{
"_index": "logstash-2014.05.06",
"_type": "syslog",
"_id": "mZvpk-_9T4WgA2zxlsxogA",
"_score": null,
"_source": {
"@version": "1",
"@timestamp": "2014-05-05T20:01:26.000-05:00",
"type": "syslog",
"syslog_pri": "163",
"syslog_program": "ubuntu",
"received_at": "2014-05-05 20:01:27 UTC",
"syslog_severity_code": 3,
"syslog_facility_code": 20,
"syslog_facility": "local4",
"syslog_severity": "error",
"@source_host": "p-aws-emmaplatformsingle01",
"@message": "trustinme",
"@host": "p-aws-emmaplatformsingle01"
},
"sort": [
1399338086000
]
}
Here is a sample from a physical machine in one of our data centers (CST) that is picked up by the central logs server (CST) - Diplays instantly/correctly
{
"_index": "logstash-2014.05.06",
"_type": "syslog",
"_id": "SjWn9aJWRGKeshylyp1j2Q",
"_score": null,
"_source": {
"@version": "1",
"@timestamp": "2014-05-06T14:01:52.000-05:00",
"type": "syslog",
"syslog_pri": "13",
"syslog_program": "teskew",
"received_at": "2014-05-06 19:01:53 UTC",
"syslog_severity_code": 5,
"syslog_facility_code": 1,
"syslog_facility": "user-level",
"syslog_severity": "notice",
"@source_host": "p-bna-apix01",
"@message": "trustinme",
"@host": "p-bna-apix01"
},
"sort": [
1399402912000
]
}
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fb88791b-1231-4db9-8888-5afd5c18d7a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.