Log stash filters log message - need help

Elastic Stack Logstash
1

Below are the log stash fields

image
image821×445 11.4 KB

Log stash error :

[2020-03-02T16:48:48,354][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"WITH CORRESPONDING LINE(S)", "HoldReason": "Invoice price exceeds purchase order price", "Supplier": "BIRLA CARBON BRASIL LTDA", "Requestor": "Curi, Henrique", "Buyer": "da Cruz, Breno", "OperatingUnit": "414_OU", "LogType": "Business_KPI", "KPI_Description": "HumanTimeEquivalent", "Process_Step": "Email-Sent", "HumanTimeEquivalent": 1200 }", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'WITH': was expecting ('true', 'false' or 'null')

Log stash pipe line config :
http://dpaste.com/198PTBC

2

That is not valid JSON, so the json filter logs an error.

Your grok filter is not anchored, so if the JSON field before HoldReason contains something that matches LOGLEVEL grok may match it. I suggest you use a different field name and do not overwrite message, then look at the original [message] field to see what is going on.

1 Like
3

i really thank you for your reply. i was even thinking in those lines but my problem is how to over come this ? as i mentioned before below is my Gork filter ,

filter {
grok {
match => { "message" => "%{LOGLEVEL:loglevel} %{GREEDYDATA:message}"}
overwrite => [ "message" ]
}

    json{
            source => "message"
    }

    if [message] != "" {
            mutate {
           		add_tag => [ "spam_log" ]
            }
    }........

and below is the value of the log field before the hold reason
field name : category
value : EMAIL BUYER WITH CORRESPONDING LINE(S)
how come this gork filter is preventing this ?
can i process the same logs again ? coz there is another important feild in the same log which i don't want to loose and because of this error these logs are not processed for months now!
i want to process these omitted logs .
i read your solution of giving a different field name , but if i change the field and lets assume it works , still i will loose the old logs !! more than everything what is problem with the field name "category" according to this gork ? is the field name of the value of the field

4

Log Levels

LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)

above is the task of the log leve lgork match. may be because of the braces ? "()"

5

The ERR?(?:OR)? part of that regexp matches the ER in BUYER.

6

investigating the same through command line. changed the output to std out , now facing

this problem !! :frowning:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.