Replace @timestamp with some other field

Elastic Stack Logstash
1

Hi,

I want to replace @timestamp with some other timestamp field having format like this 11/Jan/2014:05:06:24 +05:30. Can anybody please help?

Regards,
AjayS

2

Could you elaborate on what underlying problem you're trying to solve? Normally, reformatting and making timezone adjustments is something that the presentation layer should do, not Logstash.

3

@magnusbaeck,

Thanks for your reply and yes you are correct presentation layer should do this type of adjustment. But out of interest I want to know how to replace and match this type of timestamp.
I am having this type of timestamp entry in my log files 11/Jan/2014:05:06:24 +05:30 and I am not able to match it in filter plugin. I used "dd/mm/yy HH:mm:ss +5:30" but it is saying "_dateparsefailure" error.

I tried this but the same error

    date {
           match => { "event_time" => [ "dd/mm/yy HH:mm:ss +5:30" ] }
           target => "@timestamp"
    }

Regards,
AjayS

4

Why not use the Z token to have it parse "+5:30" as a timezone offset? Anyway, your pattern has a couple of issues:

  • There's a colon between the year and the hour, not a space.
  • 'm' means minute. Use 'M' for months.
  • Since it's the name of a month you want to match and not its number, use 'MMM'.

The following configuration example should be applicable in your case: https://www.elastic.co/guide/en/logstash/current/config-examples.html#_processing_apache_logs

5

@magnusbaeck,

Thanks. It is working now after following your suggestions. I should have checked properly.

Regards,
AjayS

6

@magnusbaeck

Hi Magnus,

I have a timestamp in the format "YYYY-MM-DD HH:mm:ss" in my logs and I want to assign this to @timestamp of logstash.

I have tried using a date filter like below:
date{

  match => [ "timestamp", "YYYY-MM-DD HH:mm:ss" ]
  
}

However, the value is not being assigned correctly. Could you please help me in this.

mytimestamp: 2016-06-13 10:00:34
assigned value to @timestamp: "2016-01-13T18:00:34.000Z"

7

As documented, the day of month is "dd" and not "DD".

8

Thanks @magnusbaeck.
Really appreciate the help.

9

Hi,
I have log file of my java application and this file contains xml. Sample xml:-
<RECORD > <EPOCH >1222.32 </EPOCH > <DATE >2016-04-25 </DATE > <TIME >18:12:09.950 </TIME > </RECORD >

My log file has many such xmls.
I want to use DATE and TIME field value of this xml as value of @timestamp in logstash.
By default @timestamp of logstash stores current date and time.
I used the help provided above but logstash is throwing warning saying:

Failed parsing date from field {:field=>"nginx_timestamp", :value=>"[EMLC_DATE] [EMLC_TIME]", :exception=>"Invalid format: "[EMLC_DATE] [EMLC_TIME]"", :config_parsers=>"yyyy-MM-dd HH:mm:ss.SSS", :config_locale=>"default=en_IN", :level=>:warn}

When I check the same in kibana than @timestamp have current date and time.

Here is my filter

filter {

  xml {
                    store_xml => "false"
                    source => "message"
                    xpath => [
                            "/RECORD/EPOCH/text()", "EMLC_EPOCH",
			"/RECORD/DATE/text()", "EMLC_DATE",
			"/RECORD/TIME/text()", "EMLC_TIME",
		]
           }
            
          
 mutate {
	strip => "%{EMLC_DATE}"
	strip => "%{EMLC_TIME}"

   		 add_field => [ "nginx_timestamp", "%{EMLC_DATE} %{EMLC_TIME}" ]
	}
	date {
   		 match => [ "nginx_timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
 }
	mutate {
   		 remove_field => [ "nginx_timestamp" ]
	}  }

I am using logstash 2.3
Can anyone help me?

10

Hi,
I have log file of my java application and this file contains xml. Sample xml:
<RECORD > <EPOCH >1222.32 </EPOCH > <DATE >2016-04-25 </DATE > <TIME >18:12:09.950 </TIME > </RECORD >

My log file has many such xmls.
I want to use DATE and TIME field value of this xml as value of @timestamp in logstash.
By default @timestamp of logstash stores current date and time.
I used the help provided above but logstash is throwing warning saying:

Failed parsing date from field {:field=>"nginx_timestamp", :value=>"[EMLC_DATE] [EMLC_TIME]", :exception=>"Invalid format: "[EMLC_DATE] [EMLC_TIME]"", :config_parsers=>"yyyy-MM-dd HH:mm:ss.SSS", :config_locale=>"default=en_IN", :level=>:warn}

When I check the same in kibana than @timestamp have current date and time.

Here is my filter

filter {

  xml {
                    store_xml => "false"
                    source => "message"
                    xpath => [
                            "/RECORD/EPOCH/text()", "EMLC_EPOCH",
			"/RECORD/DATE/text()", "EMLC_DATE",
			"/RECORD/TIME/text()", "EMLC_TIME",
		]
           }
            
          
 mutate {
	strip => "%{EMLC_DATE}"
	strip => "%{EMLC_TIME}"

   		 add_field => [ "nginx_timestamp", "%{EMLC_DATE} %{EMLC_TIME}" ]
	}
	date {
   		 match => [ "nginx_timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
 }
	mutate {
   		 remove_field => [ "nginx_timestamp" ]
	}  }

I am using logstash 2.3
Can you please help me?

11

@hearvishwas, please start a new thread for your unrelated problem. Let's not make this thread a catch-all for each and every date filter related problem.