# Grokparsefailure when field is no longer in the message

**URL:** <https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882>\
**Category:** Logstash\
**Created:** [September 18, 2017, 2:16pm UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882 "2017-09-18T14:16:07Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![scayzer65](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@scayzer65](https://discuss.elastic.co/u/scayzer65)\
**Post date:** [September 18, 2017, 2:16pm UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882/1 "2017-09-18T14:16:08Z")

</div>

Hi

I have a log that randomly shows an ip address at the end of the line. I have successfully created a filter to isolate the ip address but when the log does not produce the number I get a grokparsefailure. I really need the ip address to be isolated when it randomly appears and I was wondering what is the most efficient way to drop the grokparsefailure to stop the filter from failing?

Log passes with the following line

```
INFO UserSession [Session.4502957:Default Site:user] user logged into Default Site with protocol SFTP on 10.xxx.xxx.xxx:22 from 10.xxx.xxx.xxx:48110

```

Log fails with the following line

```
INFO SSHAuthService [Session.4502956:Default Site:user] User user can retry authentication

```

Grok

```
input{
    beats{
        port => "5044"
    }
}

filter {
    if [type] == "diagnostic" {
        grok {
            match => { "message" => "%{TIMESTAMP_ISO8601:diagstamp}%{SPACE}%{CISCO_REASON:info}%{SYSLOG5424SD:session}%{SPACE}%{USERNAME:userid}%{SPACE}%{GREEDYDATA:message}%{IPV4:client}" }
        }
        date {
           match => ["diagstamp" , "yyyy-MM-dd HH:mm:ss,SSS"]
           target => ["@timestamp"]
        }
    }
}

output {
    if [type] == "diagnostic" {
      elasticsearch {
        hosts => ["10.xxx.xxx.xxx:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}" }
      }
    }
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 18, 2017, 2:41pm UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882/2 "2017-09-18T14:41:56Z")

</div>

Just make the IP address optional with e.g. `(%{IPV4:client})?`.

---

<div class="post-metadata">

**Author:** ![scayzer65](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@scayzer65](https://discuss.elastic.co/u/scayzer65)\
**Post date:** [September 19, 2017, 1:57am UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882/3 "2017-09-19T01:57:58Z")

</div>

Hi Magnus

That works when I run the filter on the Grok Debugger but when I add the filter to the .conf file it produces a duplicate entry at the end of line so I am guessing I have a syntax error somewhere on the filter.

Input

```
2017-09-15 06:31:16,728 INFO UserSession [Session.4502959:Default Site:user] user logged into Default Site with protocol SFTP on 10.xxx.xxx.xxx:22 from 10.xxx.xxx.xxx:52167

```

Ouput

```
2017-09-15 06:31:16,728 INFO UserSession [Session.4502959:Default Site:user] user logged into Default Site with protocol SFTP on 10.xxx.xxx.xxx:22 from 10.xxx.xxx.xxx:52167 , logged into Default Site with protocol SFTP on 10.xxx.xxx.xxx:22 from 10.xxx.xxx.xxx:52167

```

.conf file

```
input{
    beats{
        port => "5044"
    }
}

filter {
    if [type] == "diagnostic" {
        grok {
            match => { "message" => "%{TIMESTAMP_ISO8601:diagstamp}%{SPACE}%{CISCO_REASON:info}%{SYSLOG5424SD:session}%{SPACE}%{USERNAME:userid}%{SPACE}%{GREEDYDATA:message} (%{IPV4:client}:%{POSINT:port})?" }
        }
        date {
           match => ["diagstamp" , "yyyy-MM-dd HH:mm:ss,SSS"]
           target => ["@timestamp"]
        }
    }
}

output {
    if [type] == "diagnostic" {
      elasticsearch {
        hosts => ["10.xxx.xxx.xxx:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}" }
      }
    }
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 19, 2017, 5:19am UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882/4 "2017-09-19T05:19:31Z")

</div>

> it produces a duplicate entry at the end of line

I don't understand. _Show_ what you get, don't _describe_ it.

What do you mean with the input and output? The lines are identical and both look like inputs.

---

<div class="post-metadata">

**Author:** ![scayzer65](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@scayzer65](https://discuss.elastic.co/u/scayzer65)\
**Post date:** [September 19, 2017, 5:51am UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882/5 "2017-09-19T05:51:52Z")

</div>

If you look at the output you will see the end of the line is replicated, which I showed in the previous reply. I have supplied the json file if that helps. The line of text from the source is not like that.

* * *

> 2017-09-15 06:31:16,728 INFO UserSession [Session.4502959:Default Site:user] user logged into Default Site with protocol SFTP on 10.xxx.xxx.xxx:22 from 10.xxx.xxx.xxx:52167 , **_logged into Default Site with protocol SFTP on 10.xxx.xxx.xxx:22 from 10.xxx.xxx.xxx:52167_**

```
    {
  "_index": "filebeat-2017.09.14",
  "_type": "diagnostic",
  "_id": "AV6Yo-KRTNMqag3Z2ImC",
  "_version": 1,
  "_score": null,
  "_source": {
    "offset": 836,
    "session": "[Session.4502959:Default Site:user]",
    "input_type": "log",
    "source": "C:\\Test\\diagnostic.log",
    "message": [
      "2017-09-15 06:31:16,728 INFO UserSession [Session.4502959:Default Site:user] user logged into Default Site with protocol SFTP on 10.xxx.xxx.xxx:22 from 10.xxx.xxx.xxx:52167 ",
      "logged into Default Site with protocol SFTP on 10.xxx.xxx.xxx:22 from 10.xxx.xxx.xxx:52167"
    ],
    "type": "diagnostic",
    "userid": "user",
    "tags": [
      "Aust_Melb",
      "beats_input_codec_plain_applied"
    ],
    "@timestamp": "2017-09-14T20:31:16.728Z",
    "@version": "1",
    "beat": {
      "hostname": "FTP1",
      "name": "FTP1",
      "version": "5.5.2"
    },
    "host": "FTP1",
    "diagstamp": "2017-09-15 06:31:16,728",
    "fields": {
      "test": "test",
      "hosts": [
        "localhost:9200"
      ]
    },
    "info": "INFO UserSession "
  },
  "fields": {
    "@timestamp": [
      1505421076728
    ]
  },
  "sort": [
    1505421076728
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 19, 2017, 5:57am UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882/6 "2017-09-19T05:57:53Z")

</div>

It looks like you are trying to [overwrite](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-overwrite) the `message` field from your grok pattern. Does the behaviour change if you use the `overwrite` parameter?

---

<div class="post-metadata">

**Author:** ![scayzer65](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@scayzer65](https://discuss.elastic.co/u/scayzer65)\
**Post date:** [September 19, 2017, 6:14am UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882/7 "2017-09-19T06:14:01Z")

</div>

I will give that a go and let you know. The strange thing is the filter works in the Grok Debugger

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 19, 2017, 7:22am UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882/8 "2017-09-19T07:22:27Z")

</div>

It looks like the pattern is capturing the correct part, I am just not sure if it is appending it to the existing message field instead of overwriting it or not.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 17, 2017, 7:22am UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882/9 "2017-10-17T07:22:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.