Capture multiple information for one match grok

[focheur91300]

Hello,
I am currently stuck on a parsing topic.
I am looking with a single match to recover several information from the same information to already capture.

I'm working on DNS log:
Jan 7 13:49:02 192.168.10.254 unbound: [33534:0] info: 192.168.1.122 proxy.gamestream.nvidia.com. A IN

I need to retrieve the following information from the same log:

• FQDN : proxy.gamestream.nvidia.com
• Domaine : nvidia.com
• TLD : com

I made 3 matches to capture each one of the information but only the first match.
Testing it individually works.

Below the current configuration :

    if "pfsense" in [tags] and [program] == "unbound" {
            grok {
                    patterns_dir => ["/usr/share/logstash/pipeline/patterns/pfsense"]
                    match => { "message" => ".*?info:\s%{IP:IP_SOURCE}\s%{HOSTNAME:FQDN}\.\s" }
                    match => { "message" => ".*?info:\s%{IP:IP_SOURCE}\s.*?\.%{DOMAIN:DOMAIN}\.\s" }
                    match => { "message" => ".*?info:\s%{IP:IP_SOURCE}\s.*?\.%{TLD:TLD}\.\s" }
                    add_tag => "pfsense-unbound"
            }
    }

Pattern :
DOMAIN [\w-]+.[\w-]+
TLD [\w-]+

Thank you in advance

[andres-perez (Andrés Pérez)]

Hi!

The main issue here is that, by default, grok will stop testing after the first match. You can alter this just by adding the corresponding option break_on_match set to false.

Another issue that may be problematic is that you should add the different patterns for the same field ("message") in one array instead of repeating multiple match options. See the example in the original documentation: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match

---

You can also obtain the same result in different ways:

• Nested grok custom patterns. Not very readable but it can be done in a single line.

• As you use a custom patterns file, you could add semantic (fields) inside the patterns and use some kind of pattern composition like the common apache log format (one of the default patterns). Not too readable either: some problem logic would be outside of the filter file.

• Start with the bigger field (FQDN) extraction in one grok and add another independent grok filter just for this field (match a couple of patterns for FQDN field). I consider it the most readable solution.

[focheur91300]

Thank you.

It worked.

    if "pfsense" in [tags] and [program] == "unbound" {
            grok {
                    patterns_dir => ["/usr/share/logstash/pipeline/patterns/pfsense"]
                    break_on_match => false
                    match => { "message" => [
                            ".*?info:\s%{IP:IP_SOURCE}\s%{FQDN:FQDN}\.\s",
                            ".*?info:\s.*?\s.*?%{DOMAIN:DOMAIN}\.\s",
                            ".*?info:\s.*?\s.*?%{TLD:TLD}\.\s" ] }
                    add_tag => "pfsense-unbound"
            }
    }