Help Filter JSON file

Elastic Stack Logstash
1

Hi, I have a record that is generated by an application, this log comes as JSON.
But the log is coming broken.

Follow Log:

(2017-06-05 11:53:22) {
"__v": 0,
"_id": "Joq3PHwo8AHiUueiZ73VXpavSo84azUg30tydFVWpQTJcvMD4ZhLLDyg9rJKFVTHOChTk4giWxBVU1V6jxTRbzJBB57J1Y4UYxO6",
"request": {
"payload": {
"addressId": "2354235235346"
},
"path": "/x2/asd/s",
"method": "post",
"headers": {
"content-length": "29",
"connection": "Keep-Alive",
"x-forwarded-server": "teste.local.local",
"x-forwarded-host": "Teste.local.local",
"incap-client-ip": "189.121.52.130",
"x-forwarded-for": "181.15.58.30, 19.57.140.8",
"incap-proxy-684": "OK",
"user-agent": "okhttp/3.6.0",
"accept-encoding": "gzip",
"content-type": "application/json; charset=UTF-8",
"origin": "CCI",
"agent": "Android;6.0;XT1097;motorola;0.27.0_homolog.CCI",
"authorization": "authorizat234ionnasdaspdkasx asda, blu me ",
"host": "meuhost.teste.teste"
}
},
"statusCode": 500,
"code": 0,
"details": "(timers.js:596:5)\n\n{"isBoom":true,"isServer":true,"data":null,"output":{"statusCode":500,"payload":{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred","code":0},"headers":{}}}",
"level": "ERROR",
"_created_at": "2017-06-05T14:53:22.840Z"

Important is the value : "message, level, details and hour"

This is my filter:

if [type] == "mobile-prod" {
      multiline {
          pattern => '^\s'
          what => "next"
          }
        }

can you help me with filter?

thanks.

2

Don't use the multiline filter, use a multiline codec. The following configuration probably works:

pattern => "^\(%{TIMESTAMP_ISO8601}\) \{$"
what => "previous"
negate => true

In other words, unless the line looks like the marker of a new logical event, join it with the preceding line.

3

I use this configuration in session {input or filter ?

Follow my file /logstash/config/logstash.conf

input {
    udp {                
        host => "0.0.0.0"
        port => 514     
        tags => "syslog"
    }                          
                                                                    
    lumberjack {                                                    
        port            => 5043                                     
        ssl_certificate => "/logstash/config/logstash-forwarder.crt"
        ssl_key         => "/logstash/config/logstash-forwarder.key"
        tags            => "lumberjack"
    }   
}                                                                                      
                                                                                       
filter {                                                                               
      if [type] == "mobile-prod" {  #this type of logstash forwarder conf in the client
          codec => multiline {                                        
              pattern => "^\(%{TIMESTAMP_ISO8601}\) \{$"              
              what => "previous"                                      
              negate => true                                          
              }                                                       
            }                                                         
output {                                                              
    elasticsearch {                                                   
        cluster  => "c4logs"                                          
        protocol => "http"                                            
        host     => "10.254.0.100"                                    
        port     => "9200"                                            
        index    => "logstash-%{+YYYY.MM.dd}"                         
    }                                                                 
}  
}

It is corret?

4

For reliable operations you need to put the multiline processing closer to the source, i.e. in Filebeat or whatever is the origin of the logs. Using a multiline filter in this way is just a bad idea.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.