Help parsing [06/Sep/2017:10:57:42 -0400]

[cchooks2 (Steven)]

I did a "last 24 hours" i was able to locate some chunks of data with no dateparsefailure, but just that showing up in the past as oppose to current. The access logs are being written in EST and are continuously coming in.

@theuntergeek

Also, what is the timezone of the machine you're on?

(UTC-06:00) Central Time (US & Canada)

Thanks

[theuntergeek (Aaron Mildenstein)]

Can you please paste the JSON from one of these entries as you see it in the Discover pane? You say that the logs are being written in EST. They should be written in Zulu notation. I would like to see the expanded JSON. Please include it as text or a screenshot, if necessary.

[cchooks2 (Steven)]

No dateparsefailure, but shows up in the past as oppose to current.

Dateparsefailure tag from today's data. Date plugin : match => ["event_timestamp" ,"dd/MMM/yyyy:HH:mm:ss"]. Have it like this because we still need to see the data coming in.

Thanks

[theuntergeek (Aaron Mildenstein)]

But it's clear that it's doing exactly what it should be doing. In the top example, I see:

"event_timestamp": "11/Sep/2017:12:15:08 -0400"

And in the same event:

"@timestamp": "2017-09-11T16:15:08.000Z"

This is exactly correct. The date filter has done what it was supposed to be doing. This is event_timestamp accurately re-configured to be in UTC. Clearly, the problem is not in Logstash, or in Elasticsearch. Both are doing exactly what they should be doing, and you've just uploaded the proof.

However...

When I look at the second example, where you have disabled the date filter and are using the @timestamp that Logstash generates for each event at ingest time, I see:

"event_timestamp": "12/Sep/2017:11:31:53 -0400"

And in the same event:

"@timestamp": "2017-09-12T18:22:33.672Z"

This is a nearly 7 hour offset. Even accounting for a 49 minute delay in ingestion—representing an offset of 6 hours—this is a huge discrepancy.

The problem could be that you're sending data, but its taking a very long time to process a backlog, so you're just not seeing the real-time data you want, due to the processing delay. This seems probable based on the data you've shared.