Failed to get Json parse failure count, stats, pipeline, plugin

  • Version: 7.3.2
  • Operating System: Ubuntu 18.04.2 LTS

Hi all,
I have a problem with failed to get json parse failure count in stats/pipelines. In my env, input comes from filebeat, when json parsing data failed due to incorrect input data format,I get json parseing error log(Error parsing json {:source=>"body", :raw=>"{"player_id":"2"u}", :exception=>#})。But when I request stats info(curl -XGET 'localhost:9600/_node/stats/pipelines?pretty'), there is no failures count. While I error occurs in other plugin(date, dissect), failure count works.

How can I get json parse failure count?

Thanks for your help!

logstash.conf:

input {                                           
beats {                                         
    port => 5044                                  
}                                               
}                                                 
                                                
filter {                                          
    dissect {                                     
    mapping => {                                
        "message" => "%{time} %{event} %{body}"   
    }                                           
    add_field => {                              
        "[@metadata][message]" => "%{message}"    
    }                                           
    remove_field => ["message"]                 
    }                                             
    date {                                        
    match => ["time", "UNIX"]                   
    remove_field => ["time"]                    
    }                                             
    json {                                        
    source => "body"                            
    target => "body"                            
    }                                             
    mutate {                                      
    rename => { "body" => "%{event}" }          
    }                                             
}                                                 
                                                
output {                                          
stdout {}                                       
}

test input data:

1559102526 EnterGame {"player_id":"1"}
1559102543 EnterGame {"player_id":"3"}
1559102545 EnterGame {"player_id":"2"}
1559102526
dfadsfadsfadsfadfad
1559102545 EnterGame {"player_id":"2"}
155910dddd2545 EnterGame {"player_id":"2"}
155910dddd2545 EnterGame {"player_id":"2"u}
1559102545 EnterGame {"player_id"fdsafdasfsdf}:"2"}

response of _node/stats/pipelines?pretty

 {
"host" : "ip-172-31-17-206",
...
        "filters" : [ {
        "id" : "0f585d5e4db725badb8b6e9f5a25c717f1c3986a96af0e8454d7ba75f0b1e293",
        "failures" : 2,
        "name" : "date",
        "events" : {
            "duration_in_millis" : 8,
            "in" : 9,
            "out" : 9
        },
        "matches" : 5
        }, {
        "id" : "04e7cfcbf5730702ffb82cc290051ef817a2c8a1e35559586d0366b698ea3447",
        "name" : "json",
        "events" : {
            "duration_in_millis" : 63,
            "in" : 9,
            "out" : 9
        }
        }, {
        "id" : "43c401ea2225daddf7125b009d98792111a30a9a2003306935c54e2072b928eb",
        "failures" : 2,
        "name" : "dissect",
        "events" : {
            "duration_in_millis" : 67,
            "in" : 9,
            "out" : 9
        },
...
}

There is code in some filters that increments these counters. I do not see any such code in the json filter.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.