Grok filter in LOGSTASH

Elastic Stack Logstash
1

Hi,
I have trouble with parsing a Tomcat LOG like :

2018-03-02 11:39:06,309| INFO|http-40074-7|com.capgemini.mvne.var.manager.simplicime.ecare.CoordonneesClientManager|Construction des données envoyées en sortie du WS: OK!

Although, my grok pattern works for this log on the Grok Debugger, the log isn't parsed.
Below is my filter and output in LOGSTASH.
Any help would be appreciated.
Thanks

filter {
    grok {
        match => [ "message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s*%{TIME}\|\s*%{LOGLEVEL:level}\|%{GREEDYDATA:tomcat_port}\|%{GREEDYDATA:requestor}\|%{GREEDYDATA:action}: %{WORD:result}" ]
    }
    if "_grokparsefailure" in [tags] {
        drop { }
    }
 }
 output {
    elasticsearch {
         hosts => ["localhost:9200"]
         index => "tomcat-%{+YYYY.MM.dd}"
   }
   stdout { codec => rubydebug }
 }
2

Please format the configuration as preformatted text, e.g. using the </> toolbar button.

3

Thank you for the formatted text tip.

4

Not sure what's up here. Build your expression gradually by starting simple, then add more and more pieces until either you're done or things break again. Also, while doing this you should avoid GREEDYDATA except for the last one. It's very inefficient and could result in false matches that can be very confusing.

5

Hi Mangus,
I did as you asked for the same log but nothing was happened.
My output on kibana remains the same :

filter {
   grok {
        match => [ "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}\s*%{TIME:time}\|\s*%{GREEDYDATA:rest_of_message}" ]
    }
   if "_grokparsefailure" in [tags] {
       drop { }
   }
}

JSON output in KIBANA is

{
  "_index": "tomcat-2018.03.02",
  "_type": "doc",
  "_id": "9P_e5mEBYJNXOGK4Snsg",
  "_version": 1,
  "_score": null,
  "_source": {
    "message": "2018-03-02 14:19:29,241| INFO|http-40074-4|com.capgemini.mvne.var.manager.simplicime.ecare.FactureManager|***getMontant Facture |0084150255||23.28***",
    "tags": [
      "_grokparsefailure",
      "_jsonparsefailure"
    ],
    "path": "/home/administrateur/log/tomcat/catalina.out",
    "@timestamp": "2018-03-02T13:19:42.392Z",
    "@version": "1",
    "host": "logstashtst"
  },
  "fields": {
    "@timestamp": [
      "2018-03-02T13:19:42.392Z"
    ]
  },
  "sort": [
    1519996782392
  ]
}
6

You didn't start simple enough. Start with ^%{YEAR:year}. Does that work? Yes? Then try ^%{YEAR:year}-%{MONTHNUM:month}.

Given your drop filter I'm surprised you get anything at all in Kibana. Are you sure you're using this Logstash configuration?

7

Should I use "^" before the %{YEAR:year} ????
In the match bloc, after "message", when should I use "," and when "=>"

It seems to me that same error happens when it is miss usage ???

Thanks

8

Should I use "^" before the %{YEAR:year} ????

Yes, if the year is the first thing on the line.

In the match bloc, after "message", when should I use "," and when "=>"

Pick one of these three syntaxes:

match => ["message", "expression"]
match => {"message" => "expression"}
match => {"message" => ["expression1", "expression2"]}

It seems to me that same error happens when it is miss usage ???

If not even ^%{YEAR:year} works then I think you're running a different configuration than you think you are. How are you starting Logstash? Any additional configuration files in /etc/logstash/conf.d or wherever you keep the files?

9

Magnus it starts working just for first part of the timestamp.
But as you can see below, I think, I'm using French timestamp. When I put
%{YEAR:year}
it parse the right side of the date which is day in my logs.

My timestamp as in the message core is like "2018-03-02 16:18:37,168" but with
match => { "message" => "%{YEAR:year }
it parse it as a day.

Could you please give me a solution ?

{
      "@version" => "1",
          "path" => "/home/administrateur/log/tomcat/catalina.out",
       "message" => "2018-03-02 16:18:37,168| INFO|http-40074-6|com.capgemini.mvne.var.manager.simplicime.ecare.FactureManager|***getMontant Facture |0085550384||23.28***",
          "host" => "lptvmsyslogngtstprd",
    "@timestamp" => 2018-03-02T15:37:41.410Z,
          "year" => "02 "
}
10

match => { "message" => "%{YEAR:year }

Your configuration can't possibly look like this because the syntax isn't valid. Try again, and format it as preformatted text so it doesn't get mangled.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.