Logstash File pattern are more , how to reduce the number of patterns

Elastic Stack Elasticsearch
1

Hi Team,

We are using logstash file , in that we are having so many Grok patterns , the lines of file is very huge , for debugging we are facing issue.
Are we having any other option ?

2

Hi Priya,

You could use below link for the information.

3

Hi shyamari,

Sample logstash file patterns

grok{
            match => {"message" => "\[%{WORD:logLevel}\] %{WORD:logType}->\|datetime:%{NOTSPACE:time}\+0000\|hostname:%{HOSTNAME:hostname}/%{IPV4:ip}\(%{IPV4}\)\|threadId:%{NOTSPACE:threadId}\|userId:%{WORD:userId}\|id:%{NOTSPACE:id}\|applicationName:%{NOTSPACE:applicationName}\|className:%{NOTSPACE:className}\|logMessage:SERVICE_DEPENDENCY: CALL-COMPLETED From %{NOTSPACE:fromService} To %{NOTSPACE:toService} on Uri %{URI:targetUri} with latency %{NUMBER:latency}"}
            add_tag => ["service_dependency"]
    }
    if "_grokparsefailure" in [tags]{
            grok{
                    remove_tag => ["_grokparsefailure"]
                    match => {"message" => "\[%{WORD:logLevel}\] %{WORD:logType}->\|datetime:%{NOTSPACE:time}\+0000\|hostname:%{HOSTNAME:hostname}/%{IPV4:ip}\(%{IPV4}\)\|threadId:%{NOTSPACE:threadId}\|userId:%{WORD:userId}\|id:%{NOTSPACE:id}\|applicationName:%{NOTSPACE:applicationName}\|className:%{NOTSPACE:className}\|logMessage:%{GREEDYDATA:logMessage}"}
                    add_tag => ["applog"]
            }
    }
	
	if "_grokparsefailure" in [tags]{
            grok{
                    remove_tag => ["_grokparsefailure"]
                    match => {"message" => "\[%{WORD:logLevel}\] %{WORD:logType}->\|datetime:%{NOTSPACE:time}\+0000\|hostname:%{HOSTNAME:hostname}/%{IPV4:ip}\(%{IPV4}\)\|threadId:%{NOTSPACE:threadId}\|id:%{NOTSPACE:id}\|className:%{NOTSPACE:className}\|logMessage:%{GREEDYDATA:logMessage}"}
                    add_tag => ["applog"]
            }
    }
	if "_grokparsefailure" in [tags]{
            grok{
                    remove_tag => ["_grokparsefailure"]
                    match => {"message" => "\[%{WORD:logLevel}\] %{WORD:logType}->\|datetime:%{NOTSPACE:time}\+0000\|hostname:%{HOSTNAME:hostname}/%{IPV4:ip}\(%{IPV4}\)\|threadId:%{NOTSPACE:threadId}\|userId:%{WORD:userId}\|id:%{NOTSPACE:id}\|applicationName:%{NOTSPACE:applicationName}\|className:%{NOTSPACE:className}"}
                    add_tag => ["applog"]
            }
    }

like this i am having so many patterns , because every log message comes with different fields.

4

You could try below by passing different log messages:

echo "2018-12-07 15:19:03" | logstash -e 'input { stdin {} } filter { date { match => [ "message", "yyyy-MM-dd HH:mm:ss"] } }'

or you could make file with logs you would like to test.

cat file.log | logstash -e 'input { stdin {} } filter { date { match => [ "message", "yyyy-MM-dd HH:mm:ss"] } }'

5

Thanks for reply,
But that was not my query , in my application i am having so many log statements , for each statement i have written one pattern.
Their is any chance to create all the fields will come in log statements , so we can least bother about the pattern is existing in logstash or not?

6

Can you provide a sample of your message?

For what I saw in your grok patterns, maybe your messages could be parsed using a KV filter, but you need to provide a sample of your message for better help.

7

Thank you so much

8

Hi ,
in logstash i have created the KV filter , application level log statements are coming as expect but i am getting below error in logstash console .
[2019-02-13T18:47:24,648][WARN ][org.logstash.FieldReference] Detected ambiguous Field Reference [I NFO] Tomcat started on port(s), which we expanded to the path [INFO, Tomcat started on port(s)];
in a future release of Logstash, ambiguous Field References will not be expanded.

This log coming from spring boot level , how to control this error.

9

It seems to me you should be using an array of patterns to match stuff, although you loose the tagging you are doing, but you can work that out another way.

grok {
  match => {
    "message" => [
      "Duration: %{NUMBER:duration}",
      "Speed: %{NUMBER:speed}"
    ]
  }
}

See
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.