How to add more than one paramter for filter

Elastic Stack Logstash
1

filter {
if [type] == "access" {
grok {
match => { "message" => "%{IPV4:ip} (?(msg "[A-Za-z ]+))" }
}
}
}

how to make it work. No matching is being done here for cause filed,

2

What's this "parameter" you're talking about? What cause field? Perhaps giving an example would make it easier to understand what you want to accomplish.

4

Okay, but what's the goal? You just want to extract the client IP address?

6 
\[msg %{QS:msg}\] \[data "%{IP:clientip}"\]
8

Yes, that's how grok expressions are used.

10

I asked for two things in the other thread and you've only provided one of them.

Also, always post configuration as preformatted text. Use markdown notation or the </> toolbar button. Unless you do this your configuration may get mangled as you post it.

11

from now own i will keep that in mind.

Configuration is there and am verifying output in kibana and it showing grokparsefailure

12 
match => { "message" => "[msg %{QS:msg}] [data "%{IP:clientip}"]" }

This isn't valid syntax so I don't think this really is what your configuration looks like. In particular, I think you have a few backslashes in there.

Configuration is there and am verifying output in kibana and it showing grokparsefailure

Please show an example event, e.g. by copying from Kibana's JSON tab.

13

Sorry this configuration doesn't make any output.

works and gives me the ip. but i want to add more fields for msg and severity.
When i try to do that its giving the desired output.
Sometimes grokparsefailure other times no output.

and 1 more thing i forgot to mention input and filter and output are in different configuration files.

14

If you don't provide what I ask for I won't be able to help.

15 
{
  "_index": "logstash-2018.09.06",
  "_type": "doc",
  "_id": "BxsBrmUBBMpmok0jOS_H",
  "_version": 1,
  "_score": null,
  "_source": {
    "prospector": {
      "type": "log"
    },
    "message": "[Thu Sep 06 14:00:28.444298 2018] [:error] [pid 16662:tid 139672042850048] [client 192.168.1.3:54909] [client 192.168.1.3] ModSecurity: Warning. Pattern match \"^[\\\\\\\\d.:]+$\" at REQUEST_HEADERS:Host. [file \"/usr/share/modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] [line \"810\"] [id \"920350\"] [rev \"2\"] [msg \"Host header is a numeric IP address\"] [data \"192.168.1.4\"] [severity \"WARNING\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"9\"] [accuracy \"9\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST\"] [tag \"WASCTC/WASC-21\"] [tag \"OWASP_TOP_10/A7\"] [tag \"PCI/6.5.10\"] [hostname \"192.168.1.4\"] [uri \"/icons/ubuntu-logo.png\"] [unique_id \"W5DlpM2Wi6dQgDWwMFAVXQAAAAU\"], referer: http://192.168.1.4/",
    "host": {
      "name": "ubuntu"
    },
    "offset": 53206,
    "type": "access",
    "beat": {
      "name": "ubuntu",
      "hostname": "ubuntu",
      "version": "6.4.0"
    },
    "input": {
      "type": "log"
    },
    "@version": "1",
    "source": "/var/log/apache2/error.log",
    "clientip": "192.168.1.3",
    "@timestamp": "2018-09-06T08:30:37.753Z",
    "tags": [
      "beats_input_codec_plain_applied"
    ]
  },
  "fields": {
    "@timestamp": [
      "2018-09-06T08:30:37.753Z"
    ]
  },
  "sort": [
    1536222637753
  ]
}

This is my json log when %{IP:clientip} is applied.
anything else wont workhere.

16

This works fine:

$ cat test.config 
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
  grok {
    match => {
      "message" => '\[msg %{QS:msg}\] \[data "%{IP:clientip}"\]'
    }
  }
}
$ cat data 
Thu Sep 06 14:00:28.444298 2018] [:error] [pid 16662:tid 139672042850048] [client 192.168.1.3:54909] [client 192.168.1.3] ModSecurity: Warning. Pattern match "^[d.:]+$" at REQUEST_HEADERS:Host. [file "/usr/share/modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "810"] [id "920350"] [rev "2"] [msg "Host header is a numeric IP address"] [data "192.168.1.4"] [severity "WARNING"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "192.168.1.4"] [uri "/icons/ubuntu-logo.png"] [unique_id "W5DlpM2Wi6dQgDWwMFAVXQAAAAU"], referer: http://192.168.1.4/", 
$ /opt/logstash/bin/logstash -f test.config < data
Settings: Default pipeline workers: 8
Pipeline main started
{
       "message" => "Thu Sep 06 14:00:28.444298 2018] [:error] [pid 16662:tid 139672042850048] [client 192.168.1.3:54909] [client 192.168.1.3] ModSecurity: Warning. Pattern match \"^[d.:]+$\" at REQUEST_HEADERS:Host. [file \"/usr/share/modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] [line \"810\"] [id \"920350\"] [rev \"2\"] [msg \"Host header is a numeric IP address\"] [data \"192.168.1.4\"] [severity \"WARNING\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"9\"] [accuracy \"9\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST\"] [tag \"WASCTC/WASC-21\"] [tag \"OWASP_TOP_10/A7\"] [tag \"PCI/6.5.10\"] [hostname \"192.168.1.4\"] [uri \"/icons/ubuntu-logo.png\"] [unique_id \"W5DlpM2Wi6dQgDWwMFAVXQAAAAU\"], referer: http://192.168.1.4/\", ",
      "@version" => "1",
    "@timestamp" => "2018-09-06T08:46:36.031Z",
          "host" => "lnxolofon",
           "msg" => "\"Host header is a numeric IP address\"",
      "clientip" => "192.168.1.4"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
17

That works on a log
[Fri Sep 07 11:47:08.881113 2018] [:error] [pid 695:tid 139704783517440] [client 192.168.1.3:50517] [client 192.168.1.3] ModSecurity: Warning. Pattern match "^[\\\\d.:]+$" at REQUEST_HEADERS:Host. [file "/usr/share/modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "792"] [id "920350"] [msg "Host header is a numeric IP address"] [data "192.168.1.12"] [severity "WARNING"] [ver "OWASP_CRS/3.1.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "192.168.1.12"] [uri "/"] [unique_id "W5IX5H8AAQEAAAK37poAAAAX"]

But don't work on the log

[Fri Sep 07 11:47:08.884151 2018] [:error] [pid 695:tid 139704783517440] [client 192.168.1.3:50517] [client 192.168.1.3] ModSecurity: Warning. detected XSS using libinjection. [file "/usr/share/modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "60"] [id "941100"] [msg "XSS Attack Detected via libinjection"] [data "Matched Data: XSS data found within ARGS_NAMES:<script>: <script>"] [severity "CRITICAL"] [ver "OWASP_CRS/3.1.0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A3"] [tag "OWASP_AppSensor/IE1"] [tag "CAPEC-242"] [hostname "192.168.1.12"] [uri "/"] [unique_id "W5IX5H8AAQEAAAK37poAAAAX"]

here am getting grokparsefailure.
The ip we are receiving here is server ip. Can change variable to client ip.
if you could help look into this also, it will most helpful.

18

I don't understand which parts of that log message you want to extract.

19

[client 192.168.1.3] and [msg "XSS Attack Detected via libinjection"] [data "Matched Data:
XSS data found within ARGS_NAMES:: "] and [severity "CRITICAL"]

20

This should work:

\[client %{IP:clientip}\] .* \[msg %{QS:msg}\] \[data %{QS:whatever}\] \[severity %{QS:severity}\]
22

It worked....
Thank you for being patience with me. I am new to here that's why it took some time for me to under stand this.
Any way thank you.

23

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.