# Logstash date extraction in logs **URL:** \ **Category:** Logstash\ **Created:** [March 8, 2019, 9:50pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563 "2019-03-08T21:50:53Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![JeanSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeansec/32/41724_2.png) [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Post date:** [March 8, 2019, 9:50pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/1 "2019-03-08T21:50:53Z")
Hello, I'm new to Elastic stack and i'm currently running an ELK stack on windows with Filebeat. I can see data in Kibana but the displayed timestamp does not correspond to the date written in the logs which is annoying for me. I tried to change the .conf file and add a filter, but that was unsuccessful. Could you explain me the procedure to solve my problem ? Is there a problem of mapping ? Here's my .conf file : # Sample Logstash configuration for creating a simple # Beats -\> Logstash -\> Elasticsearch pipeline. ``` input { beats { port => 5044 } } filter { date { match => ["logdate", "dd-MM-yyyy HH:mm:ss"] target => "@timestamp" } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" #user => "elastic" #password => "changeme" } } ``` Should I define logdate somewhere ? The format of the date in my logs is for instance : 12-05-2018 16:50:20 Thank you ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 8, 2019, 11:30pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/2 "2019-03-08T23:30:13Z")
On the JSON tab in Kibana, what does the field that contains the timestamp look like? ---
**Author:** ![Erik\_Tribou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/erik_tribou/32/40998_2.png) [@Erik\_Tribou](https://discuss.elastic.co/u/Erik_Tribou)\ **Post date:** [March 9, 2019, 3:50am UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/3 "2019-03-09T03:50:37Z")
Without a timezone in your date I believe it is being interpreted by elasticsearch as UTC instead of your local time. ---
**Author:** ![JeanSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeansec/32/41724_2.png) [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Post date:** [March 11, 2019, 2:05pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/4 "2019-03-11T14:05:06Z")
Do you talk about that ? ![1](https://us1.discourse-cdn.com/elastic/original/3X/f/5/f58462bc9df3cccf69088dd772f69c920d09f391.png) ---
**Author:** ![JeanSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeansec/32/41724_2.png) [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Post date:** [March 11, 2019, 2:09pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/5 "2019-03-11T14:09:52Z")
I'm not sure that timezone is the problem ; even with a gap in dates, my logs should be displayed on different dates since they are not from the same day. ---
**Author:** ![JeanSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeansec/32/41724_2.png) [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Post date:** [March 11, 2019, 3:43pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/6 "2019-03-11T15:43:47Z")
I found the grok debugger in Kibana to test different filters. I tried to create a custom pattern with Grok to detect my date. My sample data is : `11-05-2018 17:43:45` My filter is : > filter { > grok { > match =\> [ > "message", "%{DATESTAMP\_PERSO:m\_timestamp} " > ] > } > date { > match =\> ["m\_timestamp" , "dd-MM-yyyy HH:mm:ss"] > } > } > } And my custom pattern is : > DATESTAMP\_PERSO %{MONTHDAY}-%{MONTHNUM}-%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND} After many tries, i always get the error of empty char : > [parse\_exception] [patterns] Invalid regex pattern found in: [filter { grok { match =\> ["message", "%{DATESTAMP\_PERSO:m\_timestamp} %{LOGLEVEL:m\_level}"] } date { match =\> ["m\_timestamp" , "dd-MM-yyyy HH:mm:ss"] } } }]. empty range in char class, with { header={ processor\_type="grok" & property\_name="patterns" } } Any ideas ? ---
**Author:** ![JeanSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeansec/32/41724_2.png) [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Post date:** [March 11, 2019, 5:51pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/7 "2019-03-11T17:51:57Z")
I was wrong using the debugger , I got this working by using as input `04-08-2017 15:20:36 ERROR` and as grok pattern : `%{DATESTAMP:m_timestamp} %{LOGLEVEL:m_level}` This give me as output : { "m\_timestamp": "04-08-2017 15:20:36", "m\_level": "ERROR" } So the problem is in Logstash ? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 11, 2019, 6:36pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/8 "2019-03-11T18:36:36Z")
> [@JeanSec](#): > > So the problem is in Logstash ? It is not clear to me what you think is a problem. ---
**Author:** ![JeanSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeansec/32/41724_2.png) [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Post date:** [March 11, 2019, 6:39pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/9 "2019-03-11T18:39:37Z")
The Grok processor seems to extract the date and replace the timestamp with it but Kibana displays all the logs at the same date (today's date). But my logs are from different days ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 11, 2019, 7:11pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/10 "2019-03-11T19:11:39Z")
In Kibana, for a document that you think has the wrong @timestamp, can you show us the m\_timestamp (or logdate, if you are using that), @timestamp, and tags fields from the JSON tab? ---
**Author:** ![JeanSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeansec/32/41724_2.png) [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Post date:** [March 11, 2019, 8:12pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/11 "2019-03-11T20:12:23Z")
I don't know exactly how to access to the JSON tab but here's some screenshots that can be useful maybe (the first one comes from logstash). I can't find my m\_timestamp field in kibana. ![2](https://us1.discourse-cdn.com/elastic/original/3X/f/a/fa7b12a113c84e2f2e38532a1d609007e0e7d4a5.png) ![3](https://us1.discourse-cdn.com/elastic/original/3X/6/b/6b11cfadb5b22d3526e172b976ee670c700e7048.png) { "\_index": "filebeat-6.6.1-2019.03.11", "\_type": "doc", "\_id": "9xLrbWkBVaBf6Ws9vaal", "\_version": 1, "\_score": 1, "\_source": { "message": "13-05-2018 20:21:21 ERROR - xxxxxxxxxxxxxx", "prospector": { "type": "log" }, "host": { "name": "xxxxxxx" }, "offset": 152, "input": { "type": "log" }, "beat": { "version": "6.6.1", "name": "xxxxxxxxxxxxx", "hostname": "xxxxxxxxxxxx" }, "tags": [ "beats\_input\_codec\_plain\_applied" ], "source": "C:\Users\xxxxxx\Desktop\log\_sample\xxxxxxxxxx.log", "@version": "1", " **@timestamp": "2019-03-11T18:02:04.218Z**", "log": { "file": { "path": "C:\Users\xxx\Desktop\log\_sample\xxxx.log" } } }, "fields": { "@timestamp": [ " **2019-03-11T18:02:04.218Z**" ] } } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 11, 2019, 8:59pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/12 "2019-03-11T20:59:59Z")
The first image, which you say is from logstash, clearly shows that @timestamp is set to the value parsed from m\_timestamp, assuming you are running in UTC+0400. If you change your time picker in Kibana to cover that period (August 2017) do you see any events? Is it possible you have another filebeat that uses an elasticsearch output and bypasses logstash? ---
**Author:** ![JeanSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeansec/32/41724_2.png) [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Post date:** [March 12, 2019, 10:18pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/13 "2019-03-12T22:18:52Z")
Thank you for your reply. After several hours of tries, I think I understood the problem : I was testing my config with files that contained multilines logs. As a consequence, lines without date at the beginning created automatically new logs with current timestamp. Now, when I try with formatted logs, the result seems correct. I am now trying to treat these multilines logs. I tried with filebeat but didn't succeed, so I'm back with my local logs. I enabled the verbose mode on logstash to see the error, otherwise I can't see it and logstash shut down after saying 'successfully started". This is my config file : input { file { path =\> "C:/xxxxx/xxx.log" start\_position =\> "beginning" sincedb\_path =\> "NUL" codec =\> plain { charset =\> "ISO-8859-1" } } filter { grok { match =\> { "message", "%{DATESTAMP:m\_timestamp} %{LOGLEVEL:m\_level} - %{DATA:m\_msg} | %{DATA:m\_origin}(%{DATA:m\_file}:%{INT:m\_line:int})", "message", "\s+At %{DATA:m\_origin}(%{DATA:m\_file}:%{INT:m\_line:int}) - %{DATA:m\_msg}" } } multiline { pattern =\> "^\s+At " # '^' = beginning of line ; '\s' = Space ; '+' = 1 or more what =\> "previous" } date { match =\> ["m\_timestamp" , "dd-MM-yyyy HH:mm:ss"] } } output { elasticsearch { hosts =\> ["[http://localhost:9200](http://localhost:9200)"] index =\> "test" document\_type =\> "mytype" } stdout { codec =\> rubydebug } # Used for testing to display the results in the command prompt } **Error i got :** [ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at line 11, column 6 (byte 207) after input {\n file {\n path =\> "C:/Users/jdepeyrecave/Desktop/LOGDATE10/apui.log"\n start\_position =\> "beginning"\n sincedb\_path =\> "NUL"\n\tcodec =\> plain {\n charset =\> "ISO-8859-1"\n }\n}\nfilter {\ngrok ", :backtrace=\>["C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:49:in`compile\_graph'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in`map'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in`initialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:90:in`initialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline\_action/create.rb:43:in `block in execute'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:94:in`block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:94:in`exclusive'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline\_action/create.rb:39:in `execute'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:327:in`block in converge\_state'"]} [2019-03-12T18:01:03,297][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} PS C:\logstash-6.6.1\> ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 13, 2019, 10:53am UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/14 "2019-03-13T10:53:30Z")
> [@JeanSec](#): > > input { > file { > path =\> "C:/xxxxx/xxx.log" > start\_position =\> "beginning" > sincedb\_path =\> "NUL" > codec =\> plain { > charset =\> "ISO-8859-1" > } > } > filter { I see { for input, file, and codec, but only 2 } ---
**Author:** ![JeanSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeansec/32/41724_2.png) [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Post date:** [March 13, 2019, 1:56pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/15 "2019-03-13T13:56:55Z")
Wow ok I missed that, thank you. I found another error in my conf file, in the grok filter. The syntax with : match =\> { "message", "xxx", "message", "xxx" } was not correct. I replaced it by : grok { match =\> { "message" =\> ["%{DATESTAMP:m\_timestamp} %{LOGLEVEL:m\_level} - %{DATA:m\_msg} | %{DATA:m\_origin}(%{DATA:m\_file}:%{INT:m\_line:int})", "\s+At %{DATA:m\_origin}(%{DATA:m\_file}:%{INT:m\_line:int}) - %{DATA:m\_msg}"] } } I also modified the multilines part that was incorrect since it was trying to use the plugin instead of the codec. Here's my conf file : input { file { path =\> ["C:xxx/xxx.log"] start\_position =\> "beginning" sincedb\_path =\> "NUL" codec =\> plain { charset =\> "ISO-8859-1" } codec =\> multiline { pattern =\> "^%{DATESTAMP} " negate =\> true what =\> "previous" } } } ``` filter { grok { match => { "message" => ["%{DATESTAMP:m_timestamp} %{LOGLEVEL:m_level} - %{DATA:m_msg} \| %{DATA:m_origin}\(%{DATA:m_file}:%{INT:m_line:int}\)", "\s+At %{DATA:m_origin}\(%{DATA:m_file}:%{INT:m_line:int}\) - %{DATA:m_msg}"] } } date { match => ["m_timestamp" , "dd-MM-yyyy HH:mm:ss"] } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "test6" document_type => "mytype6" } stdout { codec => rubydebug } } ``` Still I get an error that is difficult to read, no idea for this one : [ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"NoMethodError", :message=\>"undefined method '+' for #Java::OrgLogstashConfigIrImperative::PluginStatement:0x6ad173d2", :backtrace=\>["C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:118:in `block in expr_attributes'", "org/jruby/RubyArray.java:1734:in`each'", "org/jruby/RubyEnumerable.java:936:in `inject'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:97:in`expr\_attributes'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:75:in `expr'", "org/jruby/RubyArray.java:2486:in`map'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:68:in `expr'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:47:in`block in compile'", "org/jruby/RubyArray.java:1734:in `each'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:45:in`compile'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:45:in `compile_imperative'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:49:in`compile\_graph'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in`map'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in`initialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:90:in`initialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline\_action/create.rb:43:in `block in execute'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:94:in`block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:94:in`exclusive'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline\_action/create.rb:39:in `execute'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:327:in`block in converge\_state'"]} ---
**Author:** ![JeanSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeansec/32/41724_2.png) [@JeanSec](https://discuss.elastic.co/u/JeanSec)\ **Post date:** [March 13, 2019, 2:22pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/16 "2019-03-13T14:22:36Z")
One more information : When I comment #codec =\> multiline { #pattern =\> "^%{DATESTAMP} " #negate =\> true #what =\> "previous" #} I don't get the error. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 13, 2019, 2:47pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/17 "2019-03-13T14:47:04Z")
> [@JeanSec](#): > > ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"NoMethodError", :message=\>"undefined method '+' for #Java::OrgLogstashConfigIrImperative::PluginStatement:0x6ad173d2", Wow, that's an impressively uninformative error message. You cannot have two codecs on an input. Either plain, or multiline, but not both. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 10, 2019, 2:47pm UTC](https://discuss.elastic.co/t/logstash-date-extraction-in-logs/171563/18 "2019-04-10T14:47:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.