Adding 1 day to the date

HRG · April 23, 2018, 7:24pm

I am new to Logstash and have a requirement to add 1 day to the date(Businessdate) and compare log event date with Businessdate. Can you please suggest how to achieve this.

Ex: Parsed string to extract YEAR, MONTH AND day
grok {
match => ["BusinessDate", "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}"]
}
#Setting time to 3am
mutate {
add_field => {
ExpectedTime => "03:00:00:000"
}
}

#Making date time
mutate {
add_field => {
BusinesDateTime => "%{year}-%{month}-%{day} %{ExpectedTime}"
}
}

#Convert/match to Datetime
date
{
match => ["BusinesDateTime", "YYYY-MM-DD HH:mm:ss.SSS"]
}

Now i would like to add 1 day to the "BusinessDateTime".

When i read some of existing posts, it says use "ruby" plug-in. Please let me know how to use ruby functions to add 1 day.

Can you please help how to achive this?

Jenni · April 24, 2018, 7:30am

This works:

ruby {
    code => 'event.set("@timestamp", LogStash::Timestamp.new(event.get("@timestamp")+86400))'
  }

HRG · April 24, 2018, 1:49pm

Thank you Jenni for the info. I have tried but getting below error. Can you please advise anything is missing in the code

Error
[2018-04-24T09:43:48,356][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String
[2018-04-24T09:43:48,356][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String
[2018-04-24T09:43:48,357][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String

Code
grok {
match => ["BusinessDate", "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}"]
overwrite => ["message"]
}

    if "_grokparsefailure" in [tags] {		
		mutate {				
			remove_tag => ["_grokparsefailure"]					
		}				
	}

# Set Timestamp

mutate {
	add_field => {
		ExpectedTime => "03:00:00:000"
	}
}


mutate {
	add_field => {
		BusinesDateTime => "%{year}-%{month}-%{day} %{ExpectedTime}"
	}
}
#Not required ExptectedTime in output
mutate {
	remove_field => [ExpectedTime]
}


#Convert to Date
date
{
	match => ["BusinesDateTime", "YYYY-MM-DD HH:mm:ss.SSS"]				
}

#Adding one day to BusinessDateTime

ruby {
code => 'event.set("BusinesDateTime", LogStash::Timestamp.new(event.get("BusinesDateTime")+86400))'
}

Jenni · April 24, 2018, 1:55pm

The default target of date{} is @timestamp. So your BusinesDateTime is not a LogStash::Timestamp at this point, but a string. If you want to convert BusinesDateTime to a timestamp, you have to specify this field as the target. That's probably causing the error.

HRG · April 24, 2018, 2:06pm

#Convert to Date
date
{
match => ["BusinesDateTime", "YYYY-MM-DD HH:mm:ss.SSS"]
target => "BusinesDateTime"
}

#Adding one day to BusinessDateTime

ruby {
code => 'event.set("BusinesDateTime", LogStash::Timestamp.new(event.get("BusinesDateTime")+86400))'
}

[2018-04-24T09:59:53,432][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refre
sh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>fa
lse}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}],
"properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "lati
tude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-04-24T09:59:53,460][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2018-04-24T09:59:54,268][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x67cdab run>"}
[2018-04-24T09:59:54,318][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-04-24T09:59:55,409][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String
[2018-04-24T09:59:55,426][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String
[2018-04-24T09:59:55,428][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String

Jenni · April 24, 2018, 2:29pm

Hm. Strange. Two questions:

What does the ruby debug output look like without the ruby filter? Maybe that can give us a hint.
Does this work?

ruby {
code => 'event.set("@timestamp", LogStash::Timestamp.new(Time.at(event.get("@timestamp").to_f+86400)))'
}

HRG · April 24, 2018, 2:36pm

Thank you. With this new conversion, its not giving any error but not converting date correctly.

BusinessDate BusinesDateTime(input) BusinesDateTimeConverted (After adding one day)
20180419 2018-04-19 03:00:00:000 1970-01-02T00:33:38.000Z

Am i missing anything please?. Its showing as 1970 year instead of April 20th.

Jenni · April 24, 2018, 2:40pm

There's probably something wrong with the BusinesDateTime value before the ruby filter, so the UNIX timestamp that the new Timestamp object is based on, is 0+86400 = 1970-01-01 plus one day.

... Your ExpectedTime ends with ':000', but your pattern ends with '.SSS'.

HRG · April 24, 2018, 3:02pm

Means..i should not end with "SSS". Please let me know what i should end with the milliseconds part.

Jenni · April 24, 2018, 3:15pm

There's a colon in your data, not a period.

HRG · April 24, 2018, 4:00pm

Thank you for spotting the issue Jenni. Your solution is working . Thank you.

BusinesDateTime BusinesDateTimeConverted LogEventTimeStamp
2018-01-19T08:00:00.000Z 2018-01-20T08:00:00.000Z April 19th 2018, 13:15:09.000

From the above the BusinessDateTime is : 2018-01-19 03:00:00.000

Can you please advise what to do to get to YYYY-MM-DD HH:MI:SS.SSS format. The reason is i need to compare both logEventTimeStampand BusinessDateTimeConverted

Jenni · April 24, 2018, 4:20pm

I'm afraid I don't really understand the question. Do you want to convert logEventTimeStamp to a Time object?

HRG · April 24, 2018, 8:30pm

I would like to see both "BusinessDateTimeConverted" and "LogEventTImeStamp" to "YYYY-MM-DD HH:MI;SS.SSS" format.Currently it is showing as "2018-01-19T08:00:00.000Z" format.

Jenni · April 24, 2018, 8:51pm

Business Datetime Converted doesn't really have a fixed format as it is not a string, but a Timestamp object. 2018-01-19T08:00:00.000Z is Logstash's way of displaying it. You could build a formatted string from that Timestamp again. But wouldn't Timestamp objects be more useful for comparisons?

HRG · April 25, 2018, 1:49pm

Understood. Thank you.

Is this the way we compare dates to add new filed? . Somehow its throwing error.

date {
if "LogEventTimeStamp" >= "BusinesDateTimeConverted" {
mutate {
add_field => {
LateFileStatus => "LateArrival"
}
}
} else {
mutate {
add_field => {
LateFileStatus => "On-Time"
}
}
}
}

Error

[2018-04-25T09:46:11,762][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_i
or", :message=>"Expected one of #, => at line 114, column 8 (byte 2728) after filter {\n\t\t\n\t#Match patterns based on source log origin

Line 114 is above "if" condition.

Jenni · April 25, 2018, 2:25pm

Could you post more of your configuration? The text snippet in those syntax error logs usually ends right before the error. So I'm not sure if the wrong part is even included in the code you posted.

"LogEventTimeStamp" and "BusinesDateTimeConverted" are strings. You want to compare the field values [LogEventTimeStamp] and [BusinesDateTimeConverted] (if these are both timestamps)

HRG · April 25, 2018, 2:27pm

Please see below

#Extract Business date from file

grok {
	match => ["file", "%{WORD}.%{WORD:BusinessDate}.%{GREEDYDATA:FileLastPart}"]		
}      

if "_grokparsefailure" in [tags] {		
	mutate {				
			remove_tag => ["_grokparsefailure"]		
	}				
}

#Parse Business date
	
grok {
	match => ["BusinessDate", "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}"]
}      

if "_grokparsefailure" in [tags] {		
	mutate {				
			remove_tag => ["_grokparsefailure"]					
	}				
}

# Set Timestamp

mutate {
	add_field => {
		ExpectedTime => "03:00:00.000"
	}
}


mutate {
	add_field => {
		BusinesDateTime => "%{year}-%{month}-%{day} %{ExpectedTime}"
	}
}
#Not required ExptectedTime in output
mutate {
	remove_field => [ExpectedTime]
}


#Convert to Date
date
{
	match => ["BusinesDateTime", "YYYY-MM-DD HH:mm:ss.SSS"]		
	target => "BusinesDateTime"
}

#Adding one day to BusinessDateTime

ruby {
	code => 'event.set("BusinesDateTimeConverted", LogStash::Timestamp.new(Time.at(event.get("BusinesDateTime").to_f+86400)))'				
}
	
#Remove Milliseconds	
mutate {
	gsub => ["LogEventTimeStamp", "\.\d{3}$", ""]
}


#Setting up Log timestamp to LogEventTimeStamp for the given formats.
date
{
	match => ["BusinesDateTimeConverted", "yyyy-MM-dd HH:mm:ss", "MMM dd, yyyy HH:mm:ss", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss.SSS"]		
	target => "BusinesDateTimeConverted"		
}

#Setting up Log timestamp to LogEventTimeStamp for the given formats.
date
{
	match => ["LogEventTimeStamp", "yyyy-MM-dd HH:mm:ss", "MMM dd, yyyy HH:mm:ss", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss.SSS"]		
	target => "LogEventTimeStamp"		
}

#Date comparision
date
{
if "LogEventTimeStamp >= "BusinesDateTimeConverted" {
mutate {
add_field => {
"LateFileStatus" => "LateArrival"
}
}
} else {
mutate {
add_field => {
"LateFileStatus" => "On-Time"
}
}
}
}
#End of Date Comparision

HRG · April 25, 2018, 2:28pm

yeah, i wanted to compare both timestamps.

Jenni · April 25, 2018, 2:39pm

Where is the line that says "#Match patterns based ...."? The error has to be after that.

Try if [LogEventTimeStamp] >= [BusinesDateTimeConverted] { for the comparison.

HRG · April 25, 2018, 2:40pm

#Extract Business date from file

grok {
	match => ["file", "%{WORD}.%{WORD:BusinessDate}.%{GREEDYDATA:FileLastPart}"]		
}      

if "_grokparsefailure" in [tags] {		
	mutate {				
			remove_tag => ["_grokparsefailure"]		
	}				
}

#Parse Business date
	
grok {
	match => ["BusinessDate", "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}"]
}      

if "_grokparsefailure" in [tags] {		
	mutate {				
			remove_tag => ["_grokparsefailure"]					
	}				
}

# Set Timestamp

mutate {
	add_field => {
		ExpectedTime => "03:00:00.000"
	}
}


mutate {
	add_field => {
		BusinesDateTime => "%{year}-%{month}-%{day} %{ExpectedTime}"
	}
}
#Not required ExptectedTime in output
mutate {
	remove_field => [ExpectedTime]
}


#Convert to Date
date
{
	match => ["BusinesDateTime", "YYYY-MM-DD HH:mm:ss.SSS"]		
	target => "BusinesDateTime"
}

#Adding one day to BusinessDateTime

ruby {
	code => 'event.set("BusinesDateTimeConverted", LogStash::Timestamp.new(Time.at(event.get("BusinesDateTime").to_f+86400)))'				
}
	
#Remove Milliseconds	
mutate {
	gsub => ["LogEventTimeStamp", "\.\d{3}$", ""]
}


#Setting up Log timestamp to LogEventTimeStamp for the given formats.
date
{
	match => ["BusinesDateTimeConverted", "yyyy-MM-dd HH:mm:ss", "MMM dd, yyyy HH:mm:ss", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss.SSS"]		
	target => "BusinesDateTimeConverted"		
}

#Setting up Log timestamp to LogEventTimeStamp for the given formats.
date
{
	match => ["LogEventTimeStamp", "yyyy-MM-dd HH:mm:ss", "MMM dd, yyyy HH:mm:ss", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss.SSS"]		
	target => "LogEventTimeStamp"		
}

#Date comparision
date
{
if "LogEventTimeStamp >= "BusinesDateTimeConverted" {
mutate {
add_field => {
"LateFileStatus" => "LateArrival"
}
}
} else {
mutate {
add_field => {
"LateFileStatus" => "On-Time"
}
}
}
}
#End of Date Comparision

Topic		Replies	Views
Assume the date is current date if there is no date in log Logstash	5	479	December 8, 2016
No implicit conversion of Fixnum into String when adding days to a date Logstash	3	392	April 29, 2019
How to add Date in mutate add field Logstash	11	2844	November 12, 2020
I want to add a date column to my logstash with a date i choose not a timestamp hellpp Logstash	4	301	March 24, 2020
Converting ISO8601 to date Logstash	3	1475	June 22, 2018

Adding 1 day to the date

Related topics