Adding 1 day to the date

I am new to Logstash and have a requirement to add 1 day to the date(Businessdate) and compare log event date with Businessdate. Can you please suggest how to achieve this.

Ex: Parsed string to extract YEAR, MONTH AND day
grok {
match => ["BusinessDate", "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}"]
}
#Setting time to 3am
mutate {
add_field => {
ExpectedTime => "03:00:00:000"
}
}

#Making date time
mutate {
add_field => {
BusinesDateTime => "%{year}-%{month}-%{day} %{ExpectedTime}"
}
}

#Convert/match to Datetime
date
{
match => ["BusinesDateTime", "YYYY-MM-DD HH:mm:ss.SSS"]
}

Now i would like to add 1 day to the "BusinessDateTime".

When i read some of existing posts, it says use "ruby" plug-in. Please let me know how to use ruby functions to add 1 day.

Can you please help how to achive this?

This works:

ruby {
    code => 'event.set("@timestamp", LogStash::Timestamp.new(event.get("@timestamp")+86400))'
  }
5 Likes

Thank you Jenni for the info. I have tried but getting below error. Can you please advise anything is missing in the code

Error
[2018-04-24T09:43:48,356][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String
[2018-04-24T09:43:48,356][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String
[2018-04-24T09:43:48,357][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String

Code
grok {
match => ["BusinessDate", "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}"]
overwrite => ["message"]
}

    if "_grokparsefailure" in [tags] {		
		mutate {				
			remove_tag => ["_grokparsefailure"]					
		}				
	}

# Set Timestamp

mutate {
	add_field => {
		ExpectedTime => "03:00:00:000"
	}
}


mutate {
	add_field => {
		BusinesDateTime => "%{year}-%{month}-%{day} %{ExpectedTime}"
	}
}
#Not required ExptectedTime in output
mutate {
	remove_field => [ExpectedTime]
}


#Convert to Date
date
{
	match => ["BusinesDateTime", "YYYY-MM-DD HH:mm:ss.SSS"]				
}

#Adding one day to BusinessDateTime

ruby {
code => 'event.set("BusinesDateTime", LogStash::Timestamp.new(event.get("BusinesDateTime")+86400))'
}

The default target of date{} is @timestamp. So your BusinesDateTime is not a LogStash::Timestamp at this point, but a string. If you want to convert BusinesDateTime to a timestamp, you have to specify this field as the target. That's probably causing the error.

1 Like

#Convert to Date
date
{
match => ["BusinesDateTime", "YYYY-MM-DD HH:mm:ss.SSS"]
target => "BusinesDateTime"
}

#Adding one day to BusinessDateTime

ruby {
code => 'event.set("BusinesDateTime", LogStash::Timestamp.new(event.get("BusinesDateTime")+86400))'
}

[2018-04-24T09:59:53,432][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refre
sh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>fa
lse}}}, {"string_fields"=>{"match"=>"
", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}],
"properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "lati
tude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-04-24T09:59:53,460][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2018-04-24T09:59:54,268][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x67cdab run>"}
[2018-04-24T09:59:54,318][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-04-24T09:59:55,409][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String
[2018-04-24T09:59:55,426][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String
[2018-04-24T09:59:55,428][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of Fixnum into String

Hm. Strange. Two questions:

  1. What does the ruby debug output look like without the ruby filter? Maybe that can give us a hint.

  2. Does this work?

    ruby {
    code => 'event.set("@timestamp", LogStash::Timestamp.new(Time.at(event.get("@timestamp").to_f+86400)))'
    }

Thank you. With this new conversion, its not giving any error but not converting date correctly.

BusinessDate BusinesDateTime(input) BusinesDateTimeConverted (After adding one day)
20180419 2018-04-19 03:00:00:000 1970-01-02T00:33:38.000Z

Am i missing anything please?. Its showing as 1970 year instead of April 20th.

There's probably something wrong with the BusinesDateTime value before the ruby filter, so the UNIX timestamp that the new Timestamp object is based on, is 0+86400 = 1970-01-01 plus one day.

... Your ExpectedTime ends with ':000', but your pattern ends with '.SSS'.

Means..i should not end with "SSS". Please let me know what i should end with the milliseconds part.

There's a colon in your data, not a period.

Thank you for spotting the issue Jenni. Your solution is working . Thank you.

BusinesDateTime BusinesDateTimeConverted LogEventTimeStamp
2018-01-19T08:00:00.000Z 2018-01-20T08:00:00.000Z April 19th 2018, 13:15:09.000

From the above the BusinessDateTime is : 2018-01-19 03:00:00.000

Can you please advise what to do to get to YYYY-MM-DD HH:MI:SS.SSS format. The reason is i need to compare both logEventTimeStampand BusinessDateTimeConverted

I'm afraid I don't really understand the question. Do you want to convert logEventTimeStamp to a Time object?

I would like to see both "BusinessDateTimeConverted" and "LogEventTImeStamp" to "YYYY-MM-DD HH:MI;SS.SSS" format.Currently it is showing as "2018-01-19T08:00:00.000Z" format.

Business Datetime Converted doesn't really have a fixed format as it is not a string, but a Timestamp object. 2018-01-19T08:00:00.000Z is Logstash's way of displaying it. You could build a formatted string from that Timestamp again. But wouldn't Timestamp objects be more useful for comparisons?

Understood. Thank you.

Is this the way we compare dates to add new filed? . Somehow its throwing error.

date {
if "LogEventTimeStamp" >= "BusinesDateTimeConverted" {
mutate {
add_field => {
LateFileStatus => "LateArrival"
}
}
} else {
mutate {
add_field => {
LateFileStatus => "On-Time"
}
}
}
}

Error

[2018-04-25T09:46:11,762][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_i
or", :message=>"Expected one of #, => at line 114, column 8 (byte 2728) after filter {\n\t\t\n\t#Match patterns based on source log origin

Line 114 is above "if" condition.

Could you post more of your configuration? The text snippet in those syntax error logs usually ends right before the error. So I'm not sure if the wrong part is even included in the code you posted.

"LogEventTimeStamp" and "BusinesDateTimeConverted" are strings. You want to compare the field values [LogEventTimeStamp] and [BusinesDateTimeConverted] (if these are both timestamps)

Please see below

#Extract Business date from file

grok {
	match => ["file", "%{WORD}.%{WORD:BusinessDate}.%{GREEDYDATA:FileLastPart}"]		
}      

if "_grokparsefailure" in [tags] {		
	mutate {				
			remove_tag => ["_grokparsefailure"]		
	}				
}

#Parse Business date
	
grok {
	match => ["BusinessDate", "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}"]
}      

if "_grokparsefailure" in [tags] {		
	mutate {				
			remove_tag => ["_grokparsefailure"]					
	}				
}

# Set Timestamp

mutate {
	add_field => {
		ExpectedTime => "03:00:00.000"
	}
}


mutate {
	add_field => {
		BusinesDateTime => "%{year}-%{month}-%{day} %{ExpectedTime}"
	}
}
#Not required ExptectedTime in output
mutate {
	remove_field => [ExpectedTime]
}


#Convert to Date
date
{
	match => ["BusinesDateTime", "YYYY-MM-DD HH:mm:ss.SSS"]		
	target => "BusinesDateTime"
}

#Adding one day to BusinessDateTime

ruby {
	code => 'event.set("BusinesDateTimeConverted", LogStash::Timestamp.new(Time.at(event.get("BusinesDateTime").to_f+86400)))'				
}
	
#Remove Milliseconds	
mutate {
	gsub => ["LogEventTimeStamp", "\.\d{3}$", ""]
}


#Setting up Log timestamp to LogEventTimeStamp for the given formats.
date
{
	match => ["BusinesDateTimeConverted", "yyyy-MM-dd HH:mm:ss", "MMM dd, yyyy HH:mm:ss", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss.SSS"]		
	target => "BusinesDateTimeConverted"		
}

#Setting up Log timestamp to LogEventTimeStamp for the given formats.
date
{
	match => ["LogEventTimeStamp", "yyyy-MM-dd HH:mm:ss", "MMM dd, yyyy HH:mm:ss", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss.SSS"]		
	target => "LogEventTimeStamp"		
}

#Date comparision
date
{
if "LogEventTimeStamp >= "BusinesDateTimeConverted" {
mutate {
add_field => {
"LateFileStatus" => "LateArrival"
}
}
} else {
mutate {
add_field => {
"LateFileStatus" => "On-Time"
}
}
}
}
#End of Date Comparision

yeah, i wanted to compare both timestamps.

Where is the line that says "#Match patterns based ...."? The error has to be after that.

Try if [LogEventTimeStamp] >= [BusinesDateTimeConverted] { for the comparison.

#Extract Business date from file

grok {
	match => ["file", "%{WORD}.%{WORD:BusinessDate}.%{GREEDYDATA:FileLastPart}"]		
}      

if "_grokparsefailure" in [tags] {		
	mutate {				
			remove_tag => ["_grokparsefailure"]		
	}				
}

#Parse Business date
	
grok {
	match => ["BusinessDate", "%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}"]
}      

if "_grokparsefailure" in [tags] {		
	mutate {				
			remove_tag => ["_grokparsefailure"]					
	}				
}

# Set Timestamp

mutate {
	add_field => {
		ExpectedTime => "03:00:00.000"
	}
}


mutate {
	add_field => {
		BusinesDateTime => "%{year}-%{month}-%{day} %{ExpectedTime}"
	}
}
#Not required ExptectedTime in output
mutate {
	remove_field => [ExpectedTime]
}


#Convert to Date
date
{
	match => ["BusinesDateTime", "YYYY-MM-DD HH:mm:ss.SSS"]		
	target => "BusinesDateTime"
}

#Adding one day to BusinessDateTime

ruby {
	code => 'event.set("BusinesDateTimeConverted", LogStash::Timestamp.new(Time.at(event.get("BusinesDateTime").to_f+86400)))'				
}
	
#Remove Milliseconds	
mutate {
	gsub => ["LogEventTimeStamp", "\.\d{3}$", ""]
}


#Setting up Log timestamp to LogEventTimeStamp for the given formats.
date
{
	match => ["BusinesDateTimeConverted", "yyyy-MM-dd HH:mm:ss", "MMM dd, yyyy HH:mm:ss", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss.SSS"]		
	target => "BusinesDateTimeConverted"		
}

#Setting up Log timestamp to LogEventTimeStamp for the given formats.
date
{
	match => ["LogEventTimeStamp", "yyyy-MM-dd HH:mm:ss", "MMM dd, yyyy HH:mm:ss", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss.SSS"]		
	target => "LogEventTimeStamp"		
}

#Date comparision
date
{
if "LogEventTimeStamp >= "BusinesDateTimeConverted" {
mutate {
add_field => {
"LateFileStatus" => "LateArrival"
}
}
} else {
mutate {
add_field => {
"LateFileStatus" => "On-Time"
}
}
}
}
#End of Date Comparision