# Logstash tags entries as\_grokparsefailures but multiple online debuggers says otherwise

**URL:** <https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753>\
**Category:** Logstash\
**Created:** [September 9, 2021, 8:37am UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753 "2021-09-09T08:37:00Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![stillfreem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stillfreem/32/85628_2.png) [@stillfreem](https://discuss.elastic.co/u/stillfreem)\
**Post date:** [September 9, 2021, 8:37am UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/1 "2021-09-09T08:37:00Z")

</div>

Hi All, I've a very strange question that I hope to explain eloquently.

I'm dealing with FW logs. Already wrote a grok parser that works like a charm (prefer not to provided it here, let me know if it's going to be needed?) and everything is fine, I receive the alerts as I want them in my SIEM.

However when going through the log file `/var/log/logstash/logstash-palin.log` I can see some `_grokparsefailures` such as the below (I deliberately obfuscated some of the info)

 ![grok](https://us1.discourse-cdn.com/elastic/original/3X/7/9/79495f5de3bf7ae25c3831dcea082f03f46b8ce8.jpeg)

```auto
************The log entry obfuscated*****************
+02:00 Info XXX Remove: type=LIN|proto=UDP|srcIF=|srcIP=1.1.1.1|srcPort=691|srcMAC=FF:FF:FF:FF:FF:FF|dstIP=1.1.1.1|dstPort=691|dstService=XXX-MGMT|dstIF=p1.111|rule=OP-SRV-VPN|info=Unreachable Timeout|srcNAT=1.1.1.1|dstNAT=1.1.1.1|duration=-719825|count=1|receivedBytes=1255922857|sentBytes=41005632880|receivedPackets=7757015|sentPackets=37685863|user=|protocol=|application=|target=|content=|urlcat=

```

The odd thing is that when I use an online grok debugger such [Grok Debugger](https://grokdebug.herokuapp.com/) or [This One](https://grokdebugger.com/) or [This one](https://grokconstructor.appspot.com/) the regex actually works and the log is being parsed exactly as I want. But for some reason this is still being tagged as `_grokparsefailures` in the log file and I can't see the log entries in my SIEM.

Now, one thing that comes to mind is that I have a `cidr` plugin in the logstash configuration that is checking if the source is external and if it is, it drops the entry.  
In the dropped entries above the Source IP is indeed external (actually it is Company-owned, but I am checking against a particular private network and for it it's actually external). Having said that these entries should be dropped. So is it possible that by being dropped the log entries are actually flagged as `_grokparsefailures`? And why my parser fails since multiple online grok debuggers says otherwise?

Any response will be much appreciated. Thank you.

---

<div class="post-metadata">

**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\
**Post date:** [September 9, 2021, 10:58am UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/2 "2021-09-09T10:58:24Z")

</div>

> [@stillfreem](#):
>
> So is it possible that by being dropped the log entries are actually flagged as `_grokparsefailures` ?

Yes with a code like this :

```auto
filter {
  if "_grokparsefailure" in [tags] {
    drop {}
  }
}

```

> [@stillfreem](#):
>
> And why my parser fails since multiple online grok debuggers says otherwise?

Without the grok pattern it is pretty hard to tell but did you try your pattern with a breakline at the end of the message in the grok debuggers ? Because in the error log, message contains it.

Cad

---

<div class="post-metadata">

**Author:** ![stillfreem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stillfreem/32/85628_2.png) [@stillfreem](https://discuss.elastic.co/u/stillfreem)\
**Post date:** [September 10, 2021, 9:16am UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/4 "2021-09-10T09:16:04Z")

</div>

Another entry was tagged with `_grokparsefailures`and it actually matches the grok filter???

```auto
+08:00 Security XXX Block: ???|UDP(17)|pvpn0|111:7c60:af8:5cff:efff:fffa:f691:76c|0|00:00:00:00:00:00|ff02::4d2d:5345:4152:4348:202a:2048|12112|||<no-match>|4003|(null)|(null)|0|3|0|0|0|0||||||

```

I wonder what the culprit to this might be?  
@Cad any ideas now when I posted the regex?

---

<div class="post-metadata">

**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\
**Post date:** [September 10, 2021, 8:38pm UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/5 "2021-09-10T20:38:12Z")

</div>

Hi,

Your grok is pretty hard to read, i don't see where the error came from.  
With the pattern of your data one other possibility to get informations without only using grok pattern but grok with gsub and csv plugins, i explain:

First you should do a grok to divided your data in 2 parts:

- part1 :  
`+08:00 Security XXX Block`
- part2:  
`???|UDP(17)|pvpn0|111:7c60:af8:5cff:efff:fffa:f691:76c|0|00:00:00:00:00:00|ff02::4d2d:5345:4152:4348:202a:2048|12112|||<no-match>|4003|(null)|(null)|0|3|0|0|0|0||||||`

With a pattern like this :

```auto
match => {
  "message" => "^(?<Timezone>\+[\d]{2}:[\d]{2})\s+%{WORD:LogLevel}\s+(?<OriginSource>[a-zA-Z0-9]+)\s+%{WORD:Action}:\s+%{GREEDYDATA:Informations}$"
}

```

You have 5 fields `Timezone`, `LogLevel`, `OriginSource`, `Action` and `Informations`.

And `Informations` contains  
`???|UDP(17)|pvpn0|111:7c60:af8:5cff:efff:fffa:f691:76c|0|00:00:00:00:00:00|ff02::4d2d:5345:4152:4348:202a:2048|12112|||<no-match>|4003|(null)|(null)|0|3|0|0|0|0||||||`  
But `Intormations` could be  
`type=LIN|proto=UDP|srcIF=|srcIP=1.1.1.1|srcPort=691|srcMAC=FF:FF:FF:FF:FF:FF|dstIP=1.1.1.1|dstPort=691|dstService=XXX-MGMT|dstIF=p1.111|rule=OP-SRV-VPN|info=Unreachable Timeout|srcNAT=1.1.1.1|dstNAT=1.1.1.1|duration=-719825|count=1|receivedBytes=1255922857|sentBytes=41005632880|receivedPackets=7757015|sentPackets=37685863|user=|protocol=|application=|target=|content=|urlcat=`

So `Informations` is a constant number of key-value with a pattern `key=value`. But unfortunatly you don't have a key everytime. So you should remove the `key=` with gsub filter, if it exist, and specify the key in the csv plugin.

The configuration should be like this :

```auto
filter {
  grok {
    match => {
      "message" => "^(?<Timezone>\+[\d]{2}:[\d]{2})\s+%{WORD:LogLevel}\s+(?<OriginSource>[a-zA-Z0-9]+)\s+%{WORD:Action}:\s+%{GREEDYDATA:Informations}$" 
    }
  }

  mutate {
    gsub {
      # Replace pipe and equals with something between with just a pipe (remove all keys)
      "Informations", "\|[.*]+=", "\|"
    }
  }

  csv {
    # Split on a pipe and put values in the columns
    source => "Informations"
    separator => "|"
    columns => ["TrafficType", "L4Protocol", "SrcInterface", "SourceIP", "SourcePort", "SourceMAC", "DestinationIP", "DestinationPort", "DstService", "DstIF", "Rule", "Info", "SourceNAT", "DestinationNAT", "Duration", "Count", "ReceivedBytes", "SentBytes", "ReceivedPackets", "SentPackets", "User", "L7Protocol", "Application", "Target", "Content", "URLCategory"]
  }
}

```

I hope my explanation is understandable. Tell me if it isn't.

Cad.

---

<div class="post-metadata">

**Author:** ![stillfreem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stillfreem/32/85628_2.png) [@stillfreem](https://discuss.elastic.co/u/stillfreem)\
**Post date:** [September 14, 2021, 6:24am UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/6 "2021-09-14T06:24:02Z")

</div>

Hi @Cad, this is a very good approach, never thought about it and I gave it a go but it seems the gsub does not working, ill try to narrow it down :), might also be cause some of the filed are empty, i.e. no value between ||, I included `keep_empty_captures => true` but no joy?

Otherwise, on my regex there are no errors, if you run the entry through it in an online grok debugger you'll see no errors, but for some reason in the Logstash log file, there were several entries which according to it didn't match as were tagged as `_grokparsefailures`

Thank you very much for your response again.

---

<div class="post-metadata">

**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\
**Post date:** [September 14, 2021, 6:06pm UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/7 "2021-09-14T18:06:20Z")

</div>

Hi,

Did you try with this gsub pattern `"\|[a-zA-Z]+="` instead of this `"\|[.*]+="`. I think the one i give in my last answer is to inclusive.  
The error can't came from `||` because the gsub pattern don't match it. You can try your gsub [here](https://rubular.com/) if you want.

Can you please add `stdout { codec => rubydebug }` in the output part and show us the content of one event when you have a `_grokpersefailure`.

Cad.

---

<div class="post-metadata">

**Author:** ![stillfreem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stillfreem/32/85628_2.png) [@stillfreem](https://discuss.elastic.co/u/stillfreem)\
**Post date:** [September 15, 2021, 5:13am UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/8 "2021-09-15T05:13:51Z")

</div>

It's working @Cad, many thanks for showing me this method. it was a good exercise me building the regex but this is way more elegant approach as one does not need to deal with grok failures now and then because of missing a particular character in an entry 🙂

---

<div class="post-metadata">

**Author:** ![stillfreem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stillfreem/32/85628_2.png) [@stillfreem](https://discuss.elastic.co/u/stillfreem)\
**Post date:** [September 16, 2021, 10:28am UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/9 "2021-09-16T10:28:30Z")

</div>

Hi again @Cad not sure what I did but now the events are displayed with two back slashes at the tail end

 ![123](https://us1.discourse-cdn.com/elastic/original/3X/1/1/112f9c36dfc3c94ef594fc782e0f7ca3877ed503.png)

The Config I use now is:

```auto
input
{
        stdin { }
      
}

filter
{
        grok {
                keep_empty_captures => true
                match => { "message" => "^(?<Timezone>\+[\d]{2}:[\d]{2})\s+%{WORD:LogLevel}\s+(?<OriginSource>[a-zA-Z0-9]+)\s+%{WORD:Action}:\s+%{GREEDYDATA:Informations}$" }
             }

        mutate {
                gsub => ["Informations", "\|?[a-zA-Z]+=", "\|" ]
               }

        csv {
                        source => "Informations"
                        separator => "|"
                        columns => ["TrafficType", "L4Protocol", "SrcInterface", "SourceIP", "SourcePort", "SourceMAC", "DestinationIP", "DestinationPort", "DstService", "DstIF", "Rule", "Info", "SourceNAT", "DestinationNAT", "Duration", "Count", "ReceivedBytes", "SentBytes", "ReceivedPackets", "SentPackets", "User", "L7Protocol", "Application", "Target", "Content", "URLCategory"]
            }
}
output
{
        stdout { codec => rubydebug }

```

Have you got any idea why the these dashes are there?

---

<div class="post-metadata">

**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\
**Post date:** [September 16, 2021, 10:46am UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/10 "2021-09-16T10:46:11Z")

</div>

Hi,

It's my mistake. I think you have to change `"\|"` by `"|"` in the gsub option. It's a string so we don't need to escape the pipe.

Cad.

---

<div class="post-metadata">

**Author:** ![stillfreem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stillfreem/32/85628_2.png) [@stillfreem](https://discuss.elastic.co/u/stillfreem)\
**Post date:** [September 16, 2021, 11:15am UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/11 "2021-09-16T11:15:15Z")

</div>

Working now 🙂

Just one final question. The empty captures used to be displayed as empty string `""` now its says `nil`, is that the same?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/c/1cf7ab80ee6805d59929c5d0f56c016653eeef60.png)

---

<div class="post-metadata">

**Author:** ![Cad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cad/32/86661_2.png) [@Cad](https://discuss.elastic.co/u/Cad)\
**Post date:** [September 16, 2021, 2:15pm UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/12 "2021-09-16T14:15:19Z")

</div>

It's because sometimes there is nothing between two pipes

```auto
ruby {
                code => "
                        hash = event.to_hash
                        hash.each do |k,v|
                                if v == nil
                                        event.set(k, "")
# remove it with event.remove(k)
                                end
                        end
                "
        }

```

Something like this should replace nil fields by an empty string.

Cad.

---

<div class="post-metadata">

**Author:** ![stillfreem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stillfreem/32/85628_2.png) [@stillfreem](https://discuss.elastic.co/u/stillfreem)\
**Post date:** [September 23, 2021, 2:56pm UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/13 "2021-09-23T14:56:19Z")

</div>

Apologies for the belated response @Cad , thank you again 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 21, 2021, 2:56pm UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753/14 "2021-10-21T14:56:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.