Multiples field does not work


(Ed Carlos Oliveira) #1

I send this log to logstash:

2018/10/29 16:51:14 [info] 4344#4344: *556825 ModSecurity: Warning. Matched "Operator Within' with parameter.asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .ln (138 characters omitted)' against variable TX:EXTENSION' (Value:.axd/' ) [file "/etc/nginx/modsecurity/rules/cdv.wizsolucoes.com.br_rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "971"] [id "920440"] [rev "2"] [msg "URL file extension is restricted by policy"] [data ".axd"] [severity "2"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS/POLICY/EXT_RESTRICTED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "xxx.xxx.xxx.xxx"] [uri "/ScriptResource.axd"] [unique_id "154084267418.097349"] [ref "o14,4o15,3v5,18t:urlDecodeUni,t:lowercaseo20,5"], client: xxx.xxx.xxx.xxx, server: xxx.xxxxxxxxx.com.br, request: "GET /ScriptResource.axd?d=dwY9oWetJoJoVpgL6Zq8ODw7afgDt54n3kNVwHOiwtw9HhA7PJaHLnZgRz32x1ZQ0Gi0WfZkOjQ3zpPtnCItXCiNeGVJt8pRCK8lkYpJkpNmeySNREN7YN6FadoitjJ_Q7Bcy0l86Mva-nlVLr0ixE1RHvr8VTzbPPvFP3q6m1w1&t=ffffffffad4b7194 HTTP/1.1", host: "xxx.xxxxxxxxx.com.br", referrer: "http://xxx.xxxxxxxxx.com.br/mensagem.aspx?Mensagem=5&CR=037624"

And create this patterns:

NGINX_URL referrer:\s(.+?)(",|) NGINX_SEVERITY severity\s"(.+?)("]|)
NGINX_STATUS ModSecurity:\s(.+?)(.|) NGINX_REMOTE_ADDR client:\s(.+?)(,|)
NGINX_MODSECURITY %{NGINX_STATUS:status}" ,"%{NGINX_SEVERITY:severity}" , "%{NGINX_REMOTE_ADDR:client}" , "%{NGINX_URL:url}"

My logstash conf is this:

input {
beats {
port => 5400
congestion_threshold => "40"
}
}

filter {
grok {
patterns_dir => ["/etc/logstash/patterns/"]
match => { "message" => [ "%{NGINX_STATUS:status}", "%{NGINX_SEVERITY:severity}", "%{NGINX_REMOTE_ADDR:client}", "%{NGINX_URL:url}" ] }
overwrite => [ "message" ]
}
mutate {
add_field => {"status" => "%{NGINX_STATUS}"}
add_field => {"severity" => "%{NGINX_SEVERITY}"}
add_field => {"client" => "%{NGINX_REMOTE_ADDR}"}
gsub => ["url", "referrer: ", ""]
gsub => ["client", "client: ", ""]
convert => ["response", "integer"]
convert => ["bytes", "integer"]
convert => ["responsetime", "float"]
}
geoip {
source => "clientip"
target => "geoip"
add_tag => [ "nginx-geoip" ]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
remove_field => [ "timestamp" ]
}
useragent {
source => "agent"
}
}

output {
elasticsearch {
hosts => localhost
index => "weblogs2-%{+YYYY.MM.dd}"
document_type => "nginx_logs"
}
stdout { codec => rubydebug }
}

But I do not create all the fields that I put into the grok ("% {NGINX_STATUS: status}", "% {NGINX_SEVERITY: severity}", "% {NGINX_REMOTE_ADDR: client}", "% {NGINX_URL: url}").
Only the first appears.
If I change the first filed of %{NGINX_STATUS: status} to % NGINX_URL: url} in Kibana the value of %{NGINX_URL: url} appears and not the others.

What do I have to do to display all the fields?


(Ry Biesemeyer) #2

You may be looking for the Grok Filter Plugin's break_on_match => false directive.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.