ParserError: Illegal character ((CTRL-CHAR, code 27)): only regular white space (\r, \n, \t) is allowed between tokens

Hello, I am trying to filter out a log and drop any entries that have certain IP address, however I am getting an error:

[logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"\e[32minfo\e[39m: 127.0.0.1 - - [09/Feb/2020:05:39:05 +0000] "GET /api/health HTTP/1.1" 200 - "-" "curl/7.52.1"", :exception=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 27)): only regular white space (\r, \n, \t) is allowed between tokens
at [Source: (byte)"info: 127.0.0.1 - - [09/Feb/2020:05:39:05 +0000] "GET /api/health HTTP/1.1" 200 - "-" "curl/7.52.1""; line: 1, column: 2]>}

Here is my filter:

filter {
    if ([type] == "sometype") {
            json {
                    source => "message"
                    target => "message"
            }
            if("_jsonparsefailure" in [tags]) {
                    drop {}
            } else if(![message]) {
                    drop {}
            } else if([message][req-headers][x-real-ip] == "100.000.000.001") {
                    drop {}
            }
    }

}

Sample of the log entries:

info: 127.0.0.1 - - [09/Feb/2020:05:39:00 +0000] "GET /api/health HTTP/1.1" 200 - "-" "curl/7.52.1"

info: 10.46.0.2 - - [09/Feb/2020:05:39:03 +0000] "GET /api/somelink HTTP/1.1" 200 2 "-" "curl/7.29.0"

{"name":"someapp:nodejs:request:logger","hostname":"b3049da13665","pid":72,"req_id":"6231c1d8-706f-47d1-ae61-4433bf1b57e6","level":30,"remote-address":"10.46.0.2","ip":"10.10.10.10","method":"GET","url":"/api/somelink","referer":"-","user-agent":{"family":"curl","major":"7","minor":"29","patch":"0","device":{"family":"Other","major":"0","minor":"0","patch":"0"},"os":{"family":"Other","major":"0","minor":"0","patch":"0"}},"body":{},"short-body":"{}","http-version":"1.1","response-time":5.7944,"response-hrtime":[0,5794400],"status-code":200,"req-headers":{"host":"somedomain.com","connection":"close","x-real-ip":"100.000.000.001","x-forwarded-for":"10.46.0.2, 100.000.000.001","x-forwarded-proto":"https","x-forwarded-ssl":"off","x-forwarded-port":"80","x-nginx-proxy":"true","user-agent":"curl/7.29.0","accept":"*/*"},"res-headers":{"x-frame-options":"SAMEORIGIN","x-xss-protection":"1; mode=block","x-content-type-options":"nosniff","x-download-options":"noopen","strict-transport-security":"max-age=15778476; includeSubDomains","p3p":"ABCDEF","content-security-policy":"","content-type":"text/html; charset=utf-8","content-length":"2","etag":"W/\"2-4KoCHiHd29bYzs7HHpz1ZA\"","set-cookie":["sessionId=s%3ASSZgKQhR3dCxwnSgu54Vrbtf58OyFtAz.xiksxF2G6Uz2aLtZ9DHu6oYeIQUfLiTKVaKd1IzNrCw; Path=/; Expires=Sun, 09 Feb 2020 17:39:03 GMT; HttpOnly"],"vary":"Accept-Encoding"},"req":{"method":"GET","url":"/api/somelink","headers":{"host":"somehost.com","connection":"close","x-real-ip":"104.248.190.213","x-forwarded-for":"10.46.0.2, 104.248.190.213","x-forwarded-proto":"https","x-forwarded-ssl":"off","x-forwarded-port":"80","x-nginx-proxy":"true","user-agent":"curl/7.29.0","accept":"*/*"},"remoteAddress":"172.22.0.9","remotePort":46238},"res":{"statusCode":200,"header":"HTTP/1.1 200 OK\r\nX-FRAME-OPTIONS: SAMEORIGIN\r\nX-XSS-Protection: 1; mode=block\r\nX-Content-Type-Options: nosniff\r\nX-Download-Options: noopen\r\nStrict-Transport-Security: max-age=15778476; includeSubDomains\r\nP3P: ABCDEF\r\nContent-Security-Policy: \r\nContent-Type: text/html; charset=utf-8\r\nContent-Length: 2\r\nETag: W/\"2-4KoCHiHd29bYzs7HHpz1ZA\"\r\nset-cookie: sessionId=s%3ASSZgKQhR3dCxwnSgu54Vrbtf58OyFtAz.xiksxF2G6Uz2aLtZ9DHu6oYeIQUfLiTKVaKd1IzNrCw; Path=/; Expires=Sun, 09 Feb 2020 17:39:03 GMT; HttpOnly\r\nVary: Accept-Encoding\r\nDate: Sun, 09 Feb 2020 05:39:03 GMT\r\nConnection: close\r\n\r\n"},"incoming":"<--","msg":"10.46.0.2 <-- GET /api/somelink HTTP/1.1 200 2 - curl 7.29 Other 0.0.0 5.7944 ms","time":"2020-02-09T05:39:03.249Z","v":0}
info: 127.0.0.1 - - [09/Feb/2020:05:39:05 +0000] "GET /api/health HTTP/1.1" 200 - "-" "curl/7.52.1"

Any ideas what's wrong with my filter? The first 2 conditions work it seems the 3rd one that's checking the IP is the one that throws that error.

Most of your log entries are not JSON, so the json filter will be unable to parse them.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.