Yet another @timestamp log file trouble... but it should be easy!

Elastic Stack Logstash
21

...aha... but... then, if I remove that "magic" timestamps do not match... I suspected this ruby filter stuff is a workaround, but, provide I remove it... how should I handle the 2 hours mismatch?

Best regards!

22

There is nothing to handle. There is no mismatch. @timestamp is UTC and your logs are UTC+2 and Logstash is doing the right thing. Kibana will automatically adjust all times to match the browser's timezone (but this will break if you "fix" @timestamp to be local time).

23

AHA... got it! now I understand why kibana has that "browser" correction feature!!!

Many Thanks!

24

hi Alejandro Olivan,
Can you please share the code snippet which increased @timestamp by 2:00.
I need to increase the time by 4:00.
Thanks
~ELKUser

25

What underlying problem are you trying to solve? As explained previously, @timestamp should be UTC and not local time, and if you're not getting UTC you should adjust things on the input end, i.e. typically the date filter.

26

Hi ,

My logstash config file has filter as :

date { match => ["RequestDate","MM/dd/yyyy: HH:mm:ss" ]
timezone => "UTC"
}

in My input file RequestDate is having value like :12/23/2015 3:32:00 AM ,12/25/2015 7:52:00 PM etc.

I am getting error :←[33mFailed parsing date from field {:field=>"RequestDate", :value=>"12/25/2015
8:57", :exception=>"Invalid format: "12/25/2015 8:57" is malformed at " 8:57
"", :config_parsers=>"MM/dd/yyyy: HH:mm:ss", :config_locale=>"default=en_US", :l
evel=>:warn}←[0m

Kindly help me asap.

27

Hi!

Not sure what culd be hapening there.... at first sight I think that there is some difference (remember that in order to no error to raise, patterns has to match perfect! I have had bad times whith a timestamp that had double space character between date and time in the log file!!!):
So... Maybe the problem comes because you are geting SINGLE digit hour in logfile plus AM/PM, while in your pattern you are declaring two digit hour fingerprint HH and no AM/PM) ... but this is only a guess... otherwise your pattern would be correct unlike there is some hidden space or something...

Sad I cannot help you more!
Best regards

28

First I am using csv file as an input file with code:
input {
file {
path => ["C:\Users\rajgau\Documents\Software and Tools\ExcelData\ThreeFlight.csv"]
}
}

And the Requested Date is :12/23/2015&3:32:00&AM Where & means space.

29

There are at least three problems in the date pattern:

  • There a colon after "yyyy" which isn't present in the input strings.
  • You're using a 24-hour token for the hours ("HH") instead of a 12-hour token.
  • You don't have an AM/PM token at the end of the pattern.
30

Thanks for replying magnusbaeck.
I will like to post the error again:

Error: ←[33mFailed parsing date from field {:field=>"RequestDate", :value=>"12/4/2015 1
1:54:42 PM", :exception=>"Invalid format: "12/4/2015 11:54:42 PM" is malformed
at ":42 PM"", :config_parsers=>"MM/dd/yyyy H:mm", :config_locale=>"default=en
_US", :level=>:warn}←[0m

For Example Request Date: 12/3/2015 2:52:18 AM

so there is no colon on YYYY

Can you suggest some pattern to parse the RequestDate.

31

so there is no colon on YYYY

No, not in the actual dates but according to your previous post there is a colon after the year in the date pattern:

date { match => ["RequestDate","MM/dd/yyyy: HH:mm:ss" ]
32

Now, the problem, obviously comes as I try to get the @timestamp field with my log read values, instead of the default @tiemstamp values that include the capture time data.
I have struggled with tons of combinations, leading to no success...

33

Please start a new thread for your problem, and include more details. What do your events look like, what do you want them to look like, and what's your configuration?

34

Hello,
I have this error,

{:timestamp=>"2016-09-14T16:43:48.937000+0100", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"2016-09-14 15:43:48.258000", :exception=>"Invalid format: "2016-09-14 15:43:48.258000"", :config_parsers=>"ISO8601,yyyy-MM-dd'T'HH:mm:ss.SSSZZ,yyyy-MM-dd HH:mm:ss,SSS,MMM dd YYYY HH:mm:ss", :config_locale=>"default=en_US", :level=>:warn}

my filter config"

grok {
add_tag => [ "valid" ]
match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{DATA} Processed (?:inbound|outbound) message for ([^\s]+): %{GREEDYDATA:json_data}" }
}

json {
  source => json_data
}

date {
  match => [ "timestamp","ISO8601","yyyy-MM-dd'T'HH:mm:ss.SSSZZ","yyyy-MM-dd HH:mm:ss,SSS","MMM dd YYYY HH:mm:ss" ]
  remove_field => ["timestamp"]
  target => "@timestamp"