Hello, I am trying to filter out a log and drop any entries that have certain IP address, however I am getting an error:
[logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"\e[32minfo\e[39m: 127.0.0.1 - - [09/Feb/2020:05:39:05 +0000] "GET /api/health HTTP/1.1" 200 - "-" "curl/7.52.1"", :exception=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 27)): only regular white space (\r, \n, \t) is allowed between tokens
at [Source: (byte)"info: 127.0.0.1 - - [09/Feb/2020:05:39:05 +0000] "GET /api/health HTTP/1.1" 200 - "-" "curl/7.52.1""; line: 1, column: 2]>}
Here is my filter:
filter {
if ([type] == "sometype") {
json {
source => "message"
target => "message"
}
if("_jsonparsefailure" in [tags]) {
drop {}
} else if(![message]) {
drop {}
} else if([message][req-headers][x-real-ip] == "100.000.000.001") {
drop {}
}
}
}
Sample of the log entries:
info: 127.0.0.1 - - [09/Feb/2020:05:39:00 +0000] "GET /api/health HTTP/1.1" 200 - "-" "curl/7.52.1"
info: 10.46.0.2 - - [09/Feb/2020:05:39:03 +0000] "GET /api/somelink HTTP/1.1" 200 2 "-" "curl/7.29.0"
{"name":"someapp:nodejs:request:logger","hostname":"b3049da13665","pid":72,"req_id":"6231c1d8-706f-47d1-ae61-4433bf1b57e6","level":30,"remote-address":"10.46.0.2","ip":"10.10.10.10","method":"GET","url":"/api/somelink","referer":"-","user-agent":{"family":"curl","major":"7","minor":"29","patch":"0","device":{"family":"Other","major":"0","minor":"0","patch":"0"},"os":{"family":"Other","major":"0","minor":"0","patch":"0"}},"body":{},"short-body":"{}","http-version":"1.1","response-time":5.7944,"response-hrtime":[0,5794400],"status-code":200,"req-headers":{"host":"somedomain.com","connection":"close","x-real-ip":"100.000.000.001","x-forwarded-for":"10.46.0.2, 100.000.000.001","x-forwarded-proto":"https","x-forwarded-ssl":"off","x-forwarded-port":"80","x-nginx-proxy":"true","user-agent":"curl/7.29.0","accept":"*/*"},"res-headers":{"x-frame-options":"SAMEORIGIN","x-xss-protection":"1; mode=block","x-content-type-options":"nosniff","x-download-options":"noopen","strict-transport-security":"max-age=15778476; includeSubDomains","p3p":"ABCDEF","content-security-policy":"","content-type":"text/html; charset=utf-8","content-length":"2","etag":"W/\"2-4KoCHiHd29bYzs7HHpz1ZA\"","set-cookie":["sessionId=s%3ASSZgKQhR3dCxwnSgu54Vrbtf58OyFtAz.xiksxF2G6Uz2aLtZ9DHu6oYeIQUfLiTKVaKd1IzNrCw; Path=/; Expires=Sun, 09 Feb 2020 17:39:03 GMT; HttpOnly"],"vary":"Accept-Encoding"},"req":{"method":"GET","url":"/api/somelink","headers":{"host":"somehost.com","connection":"close","x-real-ip":"104.248.190.213","x-forwarded-for":"10.46.0.2, 104.248.190.213","x-forwarded-proto":"https","x-forwarded-ssl":"off","x-forwarded-port":"80","x-nginx-proxy":"true","user-agent":"curl/7.29.0","accept":"*/*"},"remoteAddress":"172.22.0.9","remotePort":46238},"res":{"statusCode":200,"header":"HTTP/1.1 200 OK\r\nX-FRAME-OPTIONS: SAMEORIGIN\r\nX-XSS-Protection: 1; mode=block\r\nX-Content-Type-Options: nosniff\r\nX-Download-Options: noopen\r\nStrict-Transport-Security: max-age=15778476; includeSubDomains\r\nP3P: ABCDEF\r\nContent-Security-Policy: \r\nContent-Type: text/html; charset=utf-8\r\nContent-Length: 2\r\nETag: W/\"2-4KoCHiHd29bYzs7HHpz1ZA\"\r\nset-cookie: sessionId=s%3ASSZgKQhR3dCxwnSgu54Vrbtf58OyFtAz.xiksxF2G6Uz2aLtZ9DHu6oYeIQUfLiTKVaKd1IzNrCw; Path=/; Expires=Sun, 09 Feb 2020 17:39:03 GMT; HttpOnly\r\nVary: Accept-Encoding\r\nDate: Sun, 09 Feb 2020 05:39:03 GMT\r\nConnection: close\r\n\r\n"},"incoming":"<--","msg":"10.46.0.2 <-- GET /api/somelink HTTP/1.1 200 2 - curl 7.29 Other 0.0.0 5.7944 ms","time":"2020-02-09T05:39:03.249Z","v":0}
info: 127.0.0.1 - - [09/Feb/2020:05:39:05 +0000] "GET /api/health HTTP/1.1" 200 - "-" "curl/7.52.1"
Any ideas what's wrong with my filter? The first 2 conditions work it seems the 3rd one that's checking the IP is the one that throws that error.