Logstash update @timestamp

Elastic Stack Logstash
1

Hi,

I have a field that is already a datetime field:

"last_execution_time" => 2017-03-08T12:19:14.593Z,

I would like this field to be the value of @timestamp.

When I try:

		mutate {
		update => { "@timestamp" => "%{last_execution_time}" }
	}

Logstash crashes with:

06:55:20.695 [LogStash::Runner] FATAL logstash.runner - An unexpected error occurred! {:error=>#<TypeError: wrong argument type String (expected LogStash::Timestamp)

The error is clear that it is trying to use string for @timestamp but is failing, but how do I tell it not to convert it to string as it is not a string type to begin with ?

I also tried this:

		date { 
		match => [ "last_execution_time", "ISO8601" ]
		timezone => "UTC"
	}

But this does not update the @timestamp field:

"last_execution_time" => 2017-03-08T12:19:14.593Z,
"@timestamp" => 2017-03-08T12:58:20.476Z,

Thanks,
E

2

In the latter case, what error message are you getting in your log?

3

@magnusbaeck

There is no error in the log in regards to the latter case, it executes fine but the value of last_execution_time does not match @timestamp.

Any other recommendations in troubleshooting ?

Side Note: I'm using the default configuration, calling the binary directly, which logs to the screen. Then checking each line in the console for any errors.

4

That's odd. Try bumping up the log level. I'm pretty sure the date filter will log all parse errors.

5

@magnusbaeck

Although there was no error in the log as an exception. I do see the following in the tags which I just noticed:

[0] "_dateparsefailure",

Hmm, ok, so ISO8601 should match the pattern right? However, it is already a date type not a string to parse, does that make a difference with the date filter ?

Let me see what log level it is using by default and increase that maybe that will produce a better error.

6

Aha, right. Yeah, if the field is a timestamp field the date filter won't work. You probably need to use a ruby filter to assign the timestamp field to @timestamp, or make sure the field is converted to a string before you feed it to the date filter.

2 Likes
7

@magnusbaeck

Ruby ehh, ok, let me try.

8

@magnusbaeck

Ok, ruby filter is the winner. This worked.

	ruby {
		code => "event.set('@timestamp', event.get('last_execution_time'));"
	} 

"last_execution_time" => 2017-03-09T04:07:51.520Z
"@timestamp" => 2017-03-09T04:07:51.520Z

Is that ok to use or do you know of a more optimal way ?

Thanks,
E

9

That looks okay.

10

@magnusbaeck

You were very helpful, thank you!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.