Masking the password and handling two client IP's in apache_access log

Elastic Stack Logstash
1

Sometimes, I have standard apache logs starting with a single client IP and sometimes with two client IP's like this which my logstash config does not handle. Shall I use a 3rd pattern in Grok ?

1.2.3.4,5.6.7.8

Also, How to mask the value of the key secret =ABCDEF to secet=xxxxx ?

I have the below log line.
- - - [28/May/2020:21:46:07 +0200] \"GET /dev/log/foo.py?dev_name=aggr_api&aggr_name=wip_aggr_test&output_format=python&_username=_tech_user&_secret=ABCDEF HTTP/1.1\" 200 50 \"-\" \"Python-urllib/2.6"

Here is my logstash config(I have not implemented for 2 client IP's yet):

filter {
     grok {
            match => [
                "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}",
                "message" , '-*%{SYSLOG5424SD}\s":?%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}'
            ]
            overwrite => [ "message" ]
        }
        if ("&" in [request]) {
            grok {
                match => ["request", "%{DATA:script_name}\?%{GREEDYDATA:script_kv}"]
            }
            kv {
                source => "script_kv"
                field_split => "&"
                value_split => "="
                remove_char_key => "_"
                prefix => "query_string_"
            }
            mutate {
                remove_field => [ "request" ]
                remove_field => [ "script_kv" ]
            }
            if [query_string_secret] {
                mutate {
                    gsub => ["query_string_secret", ".*$", "X"]
                }
            }
        }
         if "secret" in [message] {
                mutate {
                    gsub => [
                        "message", "_secret=.*\&", "_secret=XXXX&",
                        "message", "_secret=\w+\&", "_secret=XXXX&"
                    ]
                }
            }
2

I would define a third grok pattern. Use a variant of the COMBINEDAPACHELOG pattern that replaces IPORHOST with NOTSPACE for the clientip field.

To mask the secret use mutate+replace. Either replace it with a constant, or, if necessary, replace it with a hash of the secret (created using a fingerprint filter).

1 Like
3

Okay. Thanks. Let me try this. But what I don't get is that why this is not working ?

    if "secret" in [message] {
            mutate {
                gsub => [
                    "message", "_secret=.*\&", "_secret=XXXX&",  
                    "message", "_secret=\w+\&", "_secret=XXXX&"
                ]
            }
        }

The first pattern in gsub works for the standard apache logs.It does the masking fine. Now, I added the second gsub pattern for the non-standard apahce log without the client IP and the secret masking doesn't ork with either of these patterns. Is this the right way to do it ?

4

In your sample data the value of _secret is followed by space, not &

5

Also, I ended up using DATA instead of NOTSPACE for 2 clientip's because I found spaces after the comma. So, the below works fine:

"message" , '%{DATA:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}',

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.