# Logstash filter date match not replacing @timestamp **URL:** \ **Category:** Logstash\ **Created:** [June 3, 2022, 8:17pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364 "2022-06-03T20:17:25Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 3, 2022, 8:17pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/1 "2022-06-03T20:17:25Z")
Hi, I have a bunch of PHP log to add to my ELK cluster and i cannot get the @timestamp to work. I'm pretty sure it has something to do with the config file and the date format used in the PHP logs. Here's my config file's Filter block :: ```auto filter { # ignore log comments if [message] =~ "^#" { drop {} } grok { match => ["message", "\[(?(%{MONTHDAY:day}-%{MONTH:month}-%{YEAR:year} %{TIME})) (?([\a-zA-z\s\\]*))\] (?(.|\r|\n)*)"] } date { match => ["datetime", "DD-MMM-yyyy HH:mm:ss"] target => "@timestamp" } } ``` And here's a line of log :: ```auto [03-Jun-2022 15:31:44 America/New_York] PHP Fatal error: Uncaught ArgumentCountError: Too few arguments to function KhamaiController::getTelevisionPackages(), 4 passed in /home/papi/public_html/library/shared.php on line 48 and exactly 5 expected in /home/papi/public_html/application/controllers/khamai.php:527 Stack trace: #0 /home/papi/public_html/library/shared.php(48): KhamaiController->getTelevisionPackages('96,84,103,83,86...', '1924,7054,7210,...', 'FR', '') #1 /home/papi/public_html/library/shared.php(76): callHook() #2 /home/papi/public_html/library/bootstrap.php(3): require_once('/home/papi/publ...') #3 /home/papi/public_html/index.php(22): require_once('/home/papi/publ...') #4 {main} thrown in /home/papi/public_html/application/controllers/khamai.php on line 527 ``` When i use the grok debugger i can successfully tag the datetime as one variables but it doesnt replace @timestamp. ```auto @timestamp : 03-Jun-2022 15:17:43 datetime : 06-Jul-2018 17:14:48 ``` Thanks for your help on this! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 3, 2022, 8:56pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/2 "2022-06-03T20:56:16Z")
> [@yhache](#): > > `(?(%{MONTHDAY:day}-%{MONTH:month}-%{YEAR:year} %{TIME}))` I did not know you could name fields within a custom pattern! Anyways ... it almost works for me ``` "datetime" => "03-Jun-2022 15:31:44", "@timestamp" => 2022-01-03T20:31:44Z, "month" => "Jun", ``` if you change DD (day of the year, so this forces the month to January) to dd then it does work (19 rather than 20 because January does not have DST but June does) ``` "@timestamp" => 2022-06-03T19:31:44Z, ``` Note also that since your grok has a field called 'message' the [message] field will end up as an array containing the original message and your new message string. You may want to look at the overwrite option of the filter. ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 6, 2022, 11:54am UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/3 "2022-06-06T11:54:11Z")
Hi Badger, thanks for replaying so quick! So yeah i can change the value to "dd" But even then, in your example, it was putting the @timestamp for today instead of the real one which should be 2018 (in my example) About the [message], i understand your point, i should name my last variable something else.. or not put "message" , in front of the Grok match ?! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 6, 2022, 12:42pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/4 "2022-06-06T12:42:54Z")
> [@yhache](#): > > But even then, in your example, it was putting the @timestamp for today instead of the real one which should be 2018 (in my example) No, your example was `03-Jun-2022 15:31:44 America/New_York`. ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 6, 2022, 12:47pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/5 "2022-06-06T12:47:42Z")
I'll add this Grok screenshot to help understand ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/6/d6e30d1d149d1c385672724d56676d223806954f.png) As you can see, i was able to name my variable "datetime" and the Grok parser gets it too. But even if i do the match in my filter, it does not replace @timestamp. As far as my example goes : > [@yhache](#): > > ```auto > @timestamp : 03-Jun-2022 15:17:43 > datetime : 06-Jul-2018 17:14:48 > > ``` The @timestamp took is not reflecting the right datetime (it's not the same set of dates as my screenshot but it's all the same in the end) I hope it helped. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 6, 2022, 2:24pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/6 "2022-06-06T14:24:38Z")
Can you use a `stdout` output and share the logstash output? I was not able to replicate your issue as well, for me the `date` filter is correctly replace the `@timestamp` field. ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 6, 2022, 3:08pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/7 "2022-06-06T15:08:38Z")
You know what's worst..It seems to be working for "some" of he logs : ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/9/392d3cf91731941946fecaae18b799b731ede960.png) i will continue looking into it. ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 6, 2022, 3:59pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/8 "2022-06-06T15:59:30Z")
Here's a log that was ingested correctly : ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/6/06c351b054bb3a5d9069b5d3243e0832b7b5cd4a.png) And here's what that wasn't : ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/5/55d09a41a1a45866fc7ad83a6da29a917afd7ab0.png) ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 6, 2022, 5:07pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/9 "2022-06-06T17:07:07Z")
In the second example the `date` filter failed for some reason that is still unclear. I've replaced the date in the sample message that you shared to `05-Dec-2018 18:24:16` and the following `date` filter worked without any problem: ```auto date { match => ["datetime", "dd-MMM-yyyy HH:mm:ss"] target => "@timestamp" } ``` Can you share the full line from the event where you have `_dateparsefailure`? Also, can you share other examples where the date filter failed to see if there is any pattern on them? ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 6, 2022, 5:22pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/10 "2022-06-06T17:22:46Z")
Here's the line directly from the log : ```auto [05-Dec-2018 18:24:16 America/New_York] PHP Notice: Undefined variable: langue in /home/papi/public_html/application/controllers/khamai.php on line 197 ``` Another example : ```auto [05-Dec-2018 19:12:29 America/Toronto] PHP Notice: Trying to get property of non-object in /home/papi/public_html/application/models/ipblock.php on line 28 ``` And another: ```auto [14-Aug-2018 06:31:20 America/New_York] PHP Notice: Undefined variable: langue in /home/papi/public_html/application/controllers/khamai.php on line 210 ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 6, 2022, 6:13pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/11 "2022-06-06T18:13:06Z")
It would be useful to see an event that fails to parse correctly in ``` output { stdout { codec => rubydebug } } ``` That would allow us to rule out timestamp being an array, or a LogStash::Timestamp. What is the default locale on logstash server? Although I cannot think of a locale where May would parse but Dec would not. ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 6, 2022, 7:07pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/12 "2022-06-06T19:07:06Z")
Here's the output of the stdout : ```auto "type" => "phpApi", "path" => "/home/papi/logs/php_error_log", "@version" => "1", "datetime" => "12-Dec-2018 09:30:58", "day" => "12", "message" => "[12-Dec-2018 09:30:58 America/Toronto] PHP Warning: mssql_connect(): Unable to connect to server: 10.54.2.10 in /home/papi/include/source/db/db.class_mssql.php on line 237", "month" => "Dec", "host" => "webhost01.int.cablevision.ca", "log_message" => "PHP Warning: mssql_connect(): Unable to connect to server: 10.54.2.10 in /home/papi/include/source/db/db.class_mssql.php on line 237", "time" => "09:30:58", "timezone" => "America/Toronto", "year" => "2018", "@timestamp" => 2022-06-06T19:06:25.897Z ``` ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 6, 2022, 7:17pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/13 "2022-06-06T19:17:02Z")
I tried making a new variable with the mutate function and match it after : ```auto grok { match => {"message" => "\[%{MONTHDAY:day}-%{MONTH:month}-%{YEAR:year} %{TIME:time} (?([\a-zA-z\s\\]*))\] (?(.|\r|\n)*)"} } mutate { add_field => { "newDateTime" => "%{day}-%{month}-%{year} %{time}" } } date { match => ["newDateTime", "dd-MMM-yyyy HH:mm:ss"] target => "@timestamp" } ``` But the result is the same. @timestamp never gets changed but the value of newDateTime is the good one. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 6, 2022, 7:25pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/14 "2022-06-06T19:25:40Z")
You didn't answer my question about the locale. Can you see if `mutate { gsub => ["datetime", "Dec", "Dec."] }` before the date filter helps? I am reminded of [this](https://discuss.elastic.co/t/syslog-output-date-format/264366/5) thread. ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 6, 2022, 7:41pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/15 "2022-06-06T19:41:58Z")
First, my locale is : LANG="en\_CA.UTF-8" . So It should work just fine. Secondly, here's the result of the gsub : ```auto "path" => "/home/papi/logs/php_error_log", "host" => "webhost01.int.cablevision.ca", "type" => "phpApi", "datetime" => "14-Jul.-2021 01:10:14", "day" => "14", "year" => "2021", "tags" => [ [0] "multiline" ], "@version" => "1", "message" => "DB::getConn() error : PDOException Object\n(\n [message:protected] => SQLSTATE[42000]: [Microsoft][ODBC Driver 13 for SQL Server", "month" => "Jul", "time" => "01:10:14", "timezone" => "America/New_York", "@timestamp" => 2022-06-06T19:35:34.592Z ``` I cropped the message itself beause it's multi-line. But as you can see, the @timestamp still ignores my setup in the conf file. As a reference, i have not change a thing in the conf : ```auto date { match => ["thedatetime", "dd-MMM-yyyy HH:mm:ss"] target => "@timestamp" } ``` I'm sure i am just missing something "stupid" but i don't understand what it could be. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 6, 2022, 8:02pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/16 "2022-06-06T20:02:09Z")
There is no \_dateparsefailure tag on that message. I do not understand what could be going wrong. ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 6, 2022, 8:34pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/17 "2022-06-06T20:34:32Z")
I tried different things, with a smaller data set.. Trying to see where it could fail. So i created a new field "newdatetime" then i tried to associate the datetime to it : ```auto grok { match => {"message" => "\[(?(%{MONTHDAY:day}-%{MONTH:month}-%{YEAR:year} %{TIME:time})) %{DATA:timezone}\] (?(.|\r|\n)*)"} overwrite => ["message"] } mutate{ add_field => {"newdatetime"=>""} } date { match => ["datetime","dd-MMM-yyyy HH:mm:ss"] target => "newdatetime" } ``` Here's the result... : ```auto { "type" => "phpApi", "datetime" => "08-Jul-2018 00:38:24", "@timestamp" => 2022-06-06T20:30:35.431Z, "day" => "08", "timezone" => "America/Toronto", "message" => "PHP Notice: Trying to get property of non-object in /home/papi/public_html/application/models/ipblock.php on line 28", "host" => "webhost01.int.cablevision.ca", "tags" => [ [0] "_dateparsefailure" ], "year" => "2018", "@version" => "1", "month" => "Jul", "path" => "/home/papi/logs/php_error_test_log", "time" => "00:38:24", "newdatetime" => "" } ``` It's like the match does not work at all.. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 6, 2022, 9:03pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/18 "2022-06-06T21:03:16Z")
Can you enable log.level DEBUG and see what [this line](https://github.com/logstash-plugins/logstash-filter-date/blob/cb7db481e8fe887b63006e660d7a8530e020f999/src/main/java/org/logstash/filters/DateFilter.java#L67) prints? ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 6, 2022, 9:46pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/19 "2022-06-06T21:46:17Z")
Ill try tomorrow... But i thought about something... What if i gsub all month into numbers then convert it to date time !? ---
**Author:** ![yhache](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yhache](https://discuss.elastic.co/u/yhache)\ **Post date:** [June 7, 2022, 12:26pm UTC](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364/20 "2022-06-07T12:26:14Z")
> [@Badger](#): > > `mutate { gsub => ["datetime", "Dec", "Dec."] }` [DEBUG] 2022-06-07 08:24:44.552 [Converge PipelineAction::Create] DateFilter - Date filter with format=dd-MMM-yyyy HH:mm:ss, locale=null, timezone=null built as org.logstash.filters.parser.JodaParser [Next page](https://discuss.elastic.co/t/logstash-filter-date-match-not-replacing-timestamp/306364.md?page=2)