Hello there !
I'm new to ELK and GROK filter, I look for a long time but cannot find a proper way to achieve what I want to do, so I'm asking here.
I have log coming via filebeat, the "message" field containing my original log is like this:
{"transaction":{"client_ip":"192.168.X.95","time_stamp":"Thu Apr 14 16:36:11 2022","server_id":"a4d30612b8311a3d62d1fe3224790420929e37ed","client_port":13270,"host_ip":"192.168.X.160","host_port":443,"unique_id":"1649946971","request":{"method":"GET","http_version":1.1,"uri":"/php/ajax/ajax_codification.php?type=getXXXsome URLXXX --","headers":{"Upgrade-Insecure-Requests":"1","Connection":"keep-alive","Sec-Fetch-Mode":"navigate","Accept-Encoding":"gzip, deflate, br","Cookie":"ys-gridDocCM=XXX some cockie XXX; PHPSESSID=eh0lg9fj2t1vn5frl5sbdevps2","Accept-Language":"fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3","Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8","Cache-Control":"max-age=0","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0","Sec-Fetch-Site":"none","Sec-Fetch-Dest":"document","Host":"mysite.toto.com","Sec-Fetch-User":"?1"}},"response":{"body":"<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx/1.14.0 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n","http_code":403,"headers":{"Server":"nginx/9.99.0","Date":"Thu, 14 Apr 2022 14:36:11 GMT","Content-Length":"178","Content-Type":"text/html","Connection":"keep-alive"}},"producer":{"modsecurity":"ModSecurity v3.0.6 (Linux)","connector":"ModSecurity-nginx v1.0.2","secrules_engine":"Enabled","components":["OWASP_CRS/3.4.0-dev\""]},"messages":[{"message":"SQL Injection Attack Detected via libinjection","details":{"match":"detected SQLi using libinjection.","reference":"v53,218","ruleId":"942100","file":"/usr/local/owasp-modsecurity-crs-3.3.2/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf","lineNumber":"46","data":"Matched Data: 1UEf( found within ARGS:XXX some DATA XXX--","severity":"2","ver":"OWASP_CRS/3.4.0-dev","rev":"","tags":[],"maturity":"0","accuracy":"0"}},{"message":"Detects MSSQL code execution and information gathering attempts","details":{"match":"Matched \"Operator `Rx' with parameter XXXsome paramXXX' (201 characters omitted)' against variable `ARGS:id' (Value: XXXsome valueXXX (118 characters omitted)' )","reference":"o2,12v53,218t:urlDecodeUni,t:removeCommentsChar","ruleId":"942190","file":"/usr/local/owasp-modsecurity-crs-3.3.2/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf","lineNumber":"173","data":"Matched Data: union select found within ARGS:id: 0 union select concat(XXX some SQLi injectionXXX) as valeur, 1 as ID_CODIF, 30 as OPTN, 0 as ETAT, 0 as ordre, 0 as REGROUPER from CONTACT fetch next 5 row only ","severity":"2","ver":"OWASP_CRS/3.4.0-dev","rev":"","tags":["application-multi","language-multi","platform-multi","attack-sqli","paranoia-level/1","OWASP_CRS","capec/1000/152/248/66","PCI/6.5.2"],"maturity":"0","accuracy":"0"}},{"message":"Looking for basic sql injection. Common attack string for mysql, oracle and others","details":{"match":"Matched \"Operator `Rx' with parameter `(?i)union.*?select.*?from' against variable `ARGS:id' (Value: `0 union select concat(XXXsome SQLi injectionXXX) (118 characters omitted)' )","reference":"o2,183v53,218t:urlDecodeUni","ruleId":"942270","file":"/usr/local/owasp-modsecurity-crs-3.3.2/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf","lineNumber":"291","data":"Matched Data: union select concat(XXXsome SQLi injectionXXX) as valeur, 1 as ID_CODIF, 30 as OPTN, 0 as ETAT, 0 as ordre, 0 as REGROUPER from found within ARGS:id: 0 union select concat(XXXsome SQLi injectionXXX) as valeur, 1 as ID_CODIF, 30 as OPTN, 0 as ETAT, 0 as ordre, 0 as REGROUPER from CONTACT fetch next 5 row only --","severity":"2","ver":"OWASP_CRS/3.4.0-dev","rev":"","tags":["application-multi","language-multi","platform-multi","attack-sqli","paranoia-level/1","OWASP_CRS","capec/1000/152/248/66","PCI/6.5.2"],"maturity":"0","accuracy":"0"}},{"message":"Inbound Anomaly Score Exceeded (Total Score: 20)","details":{"match":"Matched \"Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `20' )","reference":"","ruleId":"949110","file":"/usr/local/owasp-modsecurity-crs-3.3.2/rules/REQUEST-949-BLOCKING-EVALUATION.conf","lineNumber":"139","data":"","severity":"2","ver":"OWASP_CRS/3.4.0-dev","rev":"","tags":["application-multi","language-multi","platform-multi","attack-generic"],"maturity":"0","accuracy":"0"}}]}}
I successfully manage to get the field "client_ip", "time_stamp", "host_ip", "host_port", "request", "response" and "messages"
I successfully manage to break down "request" field to extract other field like: "URI", "Host", "User-Agent".
But I can't manage to break down the "messages" fields into the different "message" because I don't know how many I have with the same pattern.
The thing is, the attack I'm doing for test on my Nginx module ModSecurity trigger several rules(3) and I have several "message" with "ruleId" and attack name. But I can't know for sure how many will be triggered by an attack.
Can someone have some clue to point me to the right direction ?
Here where I am:
input {
beats {
port => 5044
}
}
filter {
# Break down my message original field to have: client_ip, time_stamp, host_ip, host_port, unique_id, request, response, messages
grok {
match => { "message" => "(?<champ_client_ip>\"client_ip\":\"%{IP}\").*?(?<x_event_time>%{DAY}\s*%{MONTH}\s*%{MONTHDAY}\s*%{TIME}\s*%{YEAR}).*?(?<champ_host_ip>\"host_ip\":\"%{IP}\").*?(?<champ_host_port>\"host_port\":[0-9]{1,5}).*?(?<champ_unique_id>\"unique_id\":\"[0-9]{1,12}\").*?(?<champ_request>\"request\":\{.*?\},).*?(?<champ_response>\"response\":\{.*?\},).*?(?<champ_messages>\"messages\":\[.*)" }
}
# Extract client IP
grok {
match => { "champ_client_ip" => "(?<x_client_ip>%{IP})" }
}
# Extract host IP
grok {
match => { "champ_host_ip" => "(?<x_host_ip>%{IP})" }
}
# Extract host_port 80 or 443
grok {
match => { "champ_host_port" => "(?<x_host_port>[0-9]{1,5})" }
}
# Extract unique_id
grok {
match => { "champ_unique_id" => "(?<x_unique_id>[0-9]{1,12})" }
}
# Break down field "request"
grok {
match => { "champ_request" => "(?<x_request_uri>%{URIPATH}).*?(?<x_request_user-agent>\"User-Agent\":.*?\",).*?(?<x_request_host>\"Host\":.*?\",)" }
}
# No break down for now
grok {
match => { "champ_response" => ".*?" }
}
# Break down "messages" field to several "message" field
grok {
#break_on_match => "false"
match => { "champ_messages" => [
"(?<x_alert_info1>\"message\":.*?\}\},)",
"(?<x_alert_info2>\"message\":.*?\}\},)",
"(?<x_alert_info3>\"message\":.*?\}\},)",
"(?<x_alert_info4>\"message\":.*?\}\},)",
"(?<x_alert_info5>\"message\":.*?\}\},)",
"(?<x_alert_info6>\"message\":.*?\}\},)"
]
}
}
}
output {
elasticsearch {
hosts => ["192.168.12.234:9200"]
manage_template => false
index => "modsec-%{+YYYY.MM}"
}
# Replace elasticsearch for debug purpose
#stdout { codec => rubydebug }
}
Best regards
