Can I add (arbitrary) new fields from inside a grok pattern?


I have been making use of freely available logstash configs and patterns, like this ( awesome one for SSHD. I've copied a small section of the file for illustrative purposes below:

# Normal
SSHD_SUCCESS        (?<sshd_result>Accepted) %{WORD:sshd_auth_type} for %{USERNAME:sshd_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{GREEDYDATA:sshd_protocol}
SSHD_DISCONNECT     Received disconnect from %{IP:sshd_client_ip}: (?<sshd_disconnect_code>\d+): %{GREEDYDATA:sshd_disconnect_reason}
SSHD_CONN_CLOSE     Connection closed by %{IP:sshd_client_ip}
SSHD_SESSION_OPEN   pam_unix\(sshd:session\): session opened for user %{USERNAME:sshd_user} by \(uid=\d+\)
SSHD_SESSION_CLOSE  pam_unix\(sshd:session\): session closed for user %{USERNAME:sshd_user}

What makes this so elegant is that we can have a single grok filter in logstash but match lots of potential log lines. This makes for a VERY clean logstash config.

Where this design lets us down is capturing the meaning/result of each of the log lines. You can see that there's an attempt in the top pattern to set the sshd_result field, and we could potentially add something similar to the rest of the patterns in the file. However, a much cleaner way would be if we could add arbitrary new fields from within the grok pattern itself. Is this already possible, and I just don't know how to do it?

An example of what I'm talking about is that if the SSHD_SUCCESS pattern matched, I'd like to create a field called 'action' with a value of 'login', and another field called 'result' with a value of 'success'. I'd like to do this (or similar) for all the patterns in the file. As far as I know, doing this at the moment requires creating a separate GROK for every single pattern in the file, which blows out the size of the code big time.

I think the 'design pattern' above is the cleanest way to do grokking en masse. I'd really like to be able to use it all the time, but this apparent limitation makes is very difficult. Any suggestions would be gratefully received!


I'm afraid you need multiple grok filters, each with an add_field option to add the fields you want.

Hi Magnus,

Is there any chance of such a feature being implemented? It would open up a whole new approach to doing large amounts of grokking, while allowing for much smaller (less error prone) logstash configs.


While convenient in special cases I'm not sure I'd be in favor of such a feature, but it's not up to me. I suggest you file a GitHub issue:

Thanks Magnus,

Will do.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.