Logstash date extraction in logs

Elastic Stack Logstash
1

Hello,
I'm new to Elastic stack and i'm currently running an ELK stack on windows with Filebeat. I can see data in Kibana but the displayed timestamp does not correspond to the date written in the logs which is annoying for me.
I tried to change the .conf file and add a filter, but that was unsuccessful. Could you explain me the procedure to solve my problem ? Is there a problem of mapping ?
Here's my .conf file :
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}
filter {
  date {
    match => [ "logdate", "dd-MM-yyyy HH:mm:ss" ]
	target => "@timestamp"
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

Should I define logdate somewhere ?
The format of the date in my logs is for instance : 12-05-2018 16:50:20
Thank you

2

On the JSON tab in Kibana, what does the field that contains the timestamp look like?

3

Without a timezone in your date I believe it is being interpreted by elasticsearch as UTC instead of your local time.

4

Do you talk about that ? 1

5

I'm not sure that timezone is the problem ; even with a gap in dates, my logs should be displayed on different dates since they are not from the same day.

6

I found the grok debugger in Kibana to test different filters.
I tried to create a custom pattern with Grok to detect my date. My sample data is :
11-05-2018 17:43:45
My filter is :

filter {
grok {
match => [
"message", "%{DATESTAMP_PERSO:m_timestamp} "
]
}
date {
match => [ "m_timestamp" , "dd-MM-yyyy HH:mm:ss" ]
}
}
}

And my custom pattern is :

DATESTAMP_PERSO %{MONTHDAY}-%{MONTHNUM}-%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}

After many tries, i always get the error of empty char :

[parse_exception] [patterns] Invalid regex pattern found in: [filter { grok { match => [ "message", "%{DATESTAMP_PERSO:m_timestamp} %{LOGLEVEL:m_level}" ] } date { match => [ "m_timestamp" , "dd-MM-yyyy HH:mm:ss" ] } } }]. empty range in char class, with { header={ processor_type="grok" & property_name="patterns" } }

Any ideas ?

7

I was wrong using the debugger , I got this working by using as input
04-08-2017 15:20:36 ERROR
and as grok pattern :
%{DATESTAMP:m_timestamp} %{LOGLEVEL:m_level}
This give me as output :
{
"m_timestamp": "04-08-2017 15:20:36",
"m_level": "ERROR"
}
So the problem is in Logstash ?

8

It is not clear to me what you think is a problem.

9

The Grok processor seems to extract the date and replace the timestamp with it but Kibana displays all the logs at the same date (today's date). But my logs are from different days

10

In Kibana, for a document that you think has the wrong @timestamp, can you show us the m_timestamp (or logdate, if you are using that), @timestamp, and tags fields from the JSON tab?

11

I don't know exactly how to access to the JSON tab but here's some screenshots that can be useful maybe (the first one comes from logstash). I can't find my m_timestamp field in kibana.

2
2.png746×322 7.11 KB
3
3.png676×565 31.8 KB

{
"_index": "filebeat-6.6.1-2019.03.11",
"_type": "doc",
"_id": "9xLrbWkBVaBf6Ws9vaal",
"_version": 1,
"_score": 1,
"_source": {
"message": "13-05-2018 20:21:21 ERROR - xxxxxxxxxxxxxx",
"prospector": {
"type": "log"
},
"host": {
"name": "xxxxxxx"
},
"offset": 152,
"input": {
"type": "log"
},
"beat": {
"version": "6.6.1",
"name": "xxxxxxxxxxxxx",
"hostname": "xxxxxxxxxxxx"
},
"tags": [
"beats_input_codec_plain_applied"
],
"source": "C:\Users\xxxxxx\Desktop\log_sample\xxxxxxxxxx.log",
"@version": "1",
"@timestamp": "2019-03-11T18:02:04.218Z",
"log": {
"file": {
"path": "C:\Users\xxx\Desktop\log_sample\xxxx.log"
}
}
},
"fields": {
"@timestamp": [
"2019-03-11T18:02:04.218Z"
]
}
}

12

The first image, which you say is from logstash, clearly shows that @timestamp is set to the value parsed from m_timestamp, assuming you are running in UTC+0400. If you change your time picker in Kibana to cover that period (August 2017) do you see any events?

Is it possible you have another filebeat that uses an elasticsearch output and bypasses logstash?

13

Thank you for your reply.
After several hours of tries, I think I understood the problem : I was testing my config with files that contained multilines logs. As a consequence, lines without date at the beginning created automatically new logs with current timestamp. Now, when I try with formatted logs, the result seems correct.
I am now trying to treat these multilines logs. I tried with filebeat but didn't succeed, so I'm back with my local logs. I enabled the verbose mode on logstash to see the error, otherwise I can't see it and logstash shut down after saying 'successfully started".
This is my config file :
input {
file {
path => "C:/xxxxx/xxx.log"
start_position => "beginning"
sincedb_path => "NUL"
codec => plain {
charset => "ISO-8859-1"
}
}
filter {
grok {
match => {
"message", "%{DATESTAMP:m_timestamp} %{LOGLEVEL:m_level} - %{DATA:m_msg} | %{DATA:m_origin}(%{DATA:m_file}:%{INT:m_line:int})",
"message", "\s+At %{DATA:m_origin}(%{DATA:m_file}:%{INT:m_line:int}) - %{DATA:m_msg}"
}
}
multiline {
pattern => "^\s+At " # '^' = beginning of line ; '\s' = Space ; '+' = 1 or more
what => "previous"
}

date {
match => [ "m_timestamp" , "dd-MM-yyyy HH:mm:ss" ]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "test"
document_type => "mytype"
}
stdout { codec => rubydebug } # Used for testing to display the results in the command prompt
}

Error i got :
[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 11, column 6 (byte 207) after input {\n file {\n path => "C:/Users/jdepeyrecave/Desktop/LOGDATE10/apui.log"\n start_position => "beginning"\n sincedb_path => "NUL"\n\tcodec => plain {\n charset => "ISO-8859-1"\n }\n}\nfilter {\ngrok ", :backtrace=>["C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:ininitialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:90:ininitialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline_action/create.rb:43:in block in execute'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:94:inblock in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in synchronize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:94:inexclusive'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline_action/create.rb:39:in execute'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:327:inblock in converge_state'"]}
[2019-03-12T18:01:03,297][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
PS C:\logstash-6.6.1>

14

I see { for input, file, and codec, but only 2 }

15

Wow ok I missed that, thank you. I found another error in my conf file, in the grok filter. The syntax with :
match => {
"message", "xxx",
"message", "xxx"
}
was not correct. I replaced it by :
grok {
match => {
"message" => ["%{DATESTAMP:m_timestamp} %{LOGLEVEL:m_level} - %{DATA:m_msg} | %{DATA:m_origin}(%{DATA:m_file}:%{INT:m_line:int})",
"\s+At %{DATA:m_origin}(%{DATA:m_file}:%{INT:m_line:int}) - %{DATA:m_msg}"]
}
}
I also modified the multilines part that was incorrect since it was trying to use the plugin instead of the codec. Here's my conf file :
input {
file {
path => ["C:xxx/xxx.log"]
start_position => "beginning"
sincedb_path => "NUL"
codec => plain {
charset => "ISO-8859-1"
}
codec => multiline {
pattern => "^%{DATESTAMP} "
negate => true
what => "previous"
}
}
}

filter {
  grok {
    match => {
      "message" => ["%{DATESTAMP:m_timestamp} %{LOGLEVEL:m_level} - %{DATA:m_msg} \| %{DATA:m_origin}\(%{DATA:m_file}:%{INT:m_line:int}\)", 
	  "\s+At %{DATA:m_origin}\(%{DATA:m_file}:%{INT:m_line:int}\) - %{DATA:m_msg}"]
    }
  }
  date {
    match => [ "m_timestamp" , "dd-MM-yyyy HH:mm:ss" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
	index => "test6"
    document_type => "mytype6"
  }
stdout { codec => rubydebug } 
}

Still I get an error that is difficult to read, no idea for this one :

[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"NoMethodError", :message=>"undefined method '+' for #Java::OrgLogstashConfigIrImperative::PluginStatement:0x6ad173d2", :backtrace=>["C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:118:in block in expr_attributes'", "org/jruby/RubyArray.java:1734:ineach'", "org/jruby/RubyEnumerable.java:936:in inject'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:97:inexpr_attributes'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:75:in expr'", "org/jruby/RubyArray.java:2486:inmap'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:68:in expr'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:47:inblock in compile'", "org/jruby/RubyArray.java:1734:in each'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler/lscl.rb:45:incompile'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:45:in compile_imperative'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "C:/logstash-6.6.1/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:ininitialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline.rb:90:ininitialize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline_action/create.rb:43:in block in execute'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:94:inblock in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in synchronize'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:94:inexclusive'", "C:/logstash-6.6.1/logstash-core/lib/logstash/pipeline_action/create.rb:39:in execute'", "C:/logstash-6.6.1/logstash-core/lib/logstash/agent.rb:327:inblock in converge_state'"]}

16

One more information :
When I comment
#codec => multiline {
#pattern => "^%{DATESTAMP} "
#negate => true
#what => "previous"
#}

I don't get the error.

17

Wow, that's an impressively uninformative error message.

You cannot have two codecs on an input. Either plain, or multiline, but not both.

1 Like
18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.