Can't replace real log time with @timestamp

Elastic Stack Logstash
1

hi
I'm trying to replace my log time with @timestamp but it doesn't work.
my log sample:
2019-07-08 01:00:13,564 INFO ir.ac.ut.sdrwebservice.SDRWebService @ batchAddStdDoc, System:G, User:25117, StudentIDs:[450188215], GroupID:4501, DocType:1349, returned 1562531413462298
and here is my logstash config:

input {
  beats {
    client_inactivity_timeout => 1200
    port => 5044
  }
}

filter {
  if ([message] !~ /batchAddStdDoc/) {
    drop { }
  }
  if ([message] !~ /returned/) {
    drop { }
  }
  grok {
    match => { "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second},%{INT:milisecond} %{LOGLEVEL:loglevel}  %{NOTSPACE:webService} @ %{NOTSPACE:function}, System:(?<systemName>.), User:%{NOTSPACE:userId}, StudentIDs:\[%{NUMBER:studentId}\], GroupID:%{GREEDYDATA:groupId}, DocType:%{NOTSPACE:docType}, returned %{INT:returnedCode}" }
  }
  date {
        locale => "en"
        match => ["message", "yyyy-MM-dd HH:mm:ss,SSS"]
        timezone => "Asia/Tehran"
        target => "@timestamp"
        add_field => { "debug" => "timestampMatched"}
   }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

thanks for your help.

2

Hi Mohammad,

The problem is that you're parsing the whole message to create the timestamp:

match => ["message", "yyyy-MM-dd HH:mm:ss,SSS"]

Try to grok the date with the following pattern:

%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} ...

and use it with the date plugin:

match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"]

Fyi: target => "@timestamp" is not necessary (by default)

3

thanks for your help.
I changed my filter like this:

filter {
  if ([message] !~ /batchAddStdDoc/) {
    drop { }
  }
  if ([message] !~ /returned/) {
    drop { }
  }
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE:webService} @ %{NOTSPACE:function}, System:(?<systemName>.), User:%{NOTSPACE:userId}, StudentIDs:\[%{NUMBER:studentId}\], GroupID:%{GREEDYDATA:groupId}, DocType:%{NOTSPACE:docType}, returned %{INT:returnedCode}" }
  }
  date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"]
   }
}

but it's not working anymore and nothing in Discover part of Kibana!

4

I'm one step forward now. when I use this filter it works:

filter {
  if ([message] !~ /batchAddStdDoc/) {
    drop { }
  }
  if ([message] !~ /returned/) {
    drop { }
  }
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}  %{NOTSPACE:webService} @ %{NOTSPACE:function}, System:(?<systemName>.), User:%{NOTSPACE:userId}, StudentIDs:\[%{NUMBER:studentId}\], GroupID:%{GREEDYDATA:groupId}, DocType:%{NOTSPACE:docType}, returned %{INT:returnedCode}" }
  }

but when I use date filter it doesn't work anymore and nothing to Discover with Kibana!

filter {
  if ([message] !~ /batchAddStdDoc/) {
    drop { }
  }
  if ([message] !~ /returned/) {
    drop { }
  }
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}  %{NOTSPACE:webService} @ %{NOTSPACE:function}, System:(?<systemName>.), User:%{NOTSPACE:userId}, StudentIDs:\[%{NUMBER:studentId}\], GroupID:%{GREEDYDATA:groupId}, DocType:%{NOTSPACE:docType}, returned %{INT:returnedCode}" }
  }
  date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"]
    timezone => "Asia/Tehran"
    target => "@timestamp"
  }
}

do you have any idea @Rom1?

5

In your grok pattern you have two spaces between

%{LOGLEVEL:loglevel}  %{NOTSPACE:webService}

However, your message only has one, so your get a _grokparsefailure. Remove one of the spaces and you will get

   "timestamp" => "2019-07-08 01:00:13,564",
    "loglevel" => "INFO",

etc.

6

thanks Badger.
my logs has 2 space there but I don't know why it's not visible here!
finally, I solved It. my filter was OK and I need to change time range because my logs were at least for two days ago and I was looking for them in today time range in Kibana.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.