Hi,
I have been making use of freely available logstash configs and patterns, like this (https://github.com/nxhack/logstash/blob/master/patterns/sshd) awesome one for SSHD. I've copied a small section of the file for illustrative purposes below:
# Normal
SSHD_SUCCESS (?<sshd_result>Accepted) %{WORD:sshd_auth_type} for %{USERNAME:sshd_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{GREEDYDATA:sshd_protocol}
SSHD_DISCONNECT Received disconnect from %{IP:sshd_client_ip}: (?<sshd_disconnect_code>\d+): %{GREEDYDATA:sshd_disconnect_reason}
SSHD_CONN_CLOSE Connection closed by %{IP:sshd_client_ip}
SSHD_SESSION_OPEN pam_unix\(sshd:session\): session opened for user %{USERNAME:sshd_user} by \(uid=\d+\)
SSHD_SESSION_CLOSE pam_unix\(sshd:session\): session closed for user %{USERNAME:sshd_user}
SSHD_NORMAL_LOG %{SSHD_SUCCESS}|%{SSHD_DISCONNECT}|%{SSHD_CONN_CLOSE}|%{SSHD_SESSION_OPEN}|%{SSHD_SESSION_CLOSE}
What makes this so elegant is that we can have a single grok filter in logstash but match lots of potential log lines. This makes for a VERY clean logstash config.
Where this design lets us down is capturing the meaning/result of each of the log lines. You can see that there's an attempt in the top pattern to set the sshd_result field, and we could potentially add something similar to the rest of the patterns in the file. However, a much cleaner way would be if we could add arbitrary new fields from within the grok pattern itself. Is this already possible, and I just don't know how to do it?
An example of what I'm talking about is that if the SSHD_SUCCESS pattern matched, I'd like to create a field called 'action' with a value of 'login', and another field called 'result' with a value of 'success'. I'd like to do this (or similar) for all the patterns in the file. As far as I know, doing this at the moment requires creating a separate GROK for every single pattern in the file, which blows out the size of the code big time.
I think the 'design pattern' above is the cleanest way to do grokking en masse. I'd really like to be able to use it all the time, but this apparent limitation makes is very difficult. Any suggestions would be gratefully received!
Regards,
Nick