Logstash Field Types Not Working Intermittently

Elastic Stack Logstash
1

Hi Guys, I have seen this type of issue once or twice before but i can't seem to get it sorted my side still.

I have a field defined as an INT in my patterns, which today has thrown out the error (in Kibana):

Visualize: Expected numeric type on field [MS], but got [string]

I understand that for some reason Kibana is seeing this value as a string instead of an int as expected hence the error, but without a single config change to my patterns or Logstash filters i don't see why this would suddenly change.

Here is the logstash config it is passing through:

filter {
      if [type] == "swAuditLog" {
        grok {
            patterns_dir => ["/opt/logstash/patterns"]
            match => { "message" => "%{SW_AUDIT}" }
        }
        
        if "_grokparsefailure" in [tags] {
            drop { }
        }

        date {
            timezone => "Africa/Harare"
            match => ["LoggedDateTime", "YYYY/MM/dd HH:mm:ss"]
            target => "@timestamp"
        }
        
        mutate {
            remove_tag => [ "beats_input_codec_plain_applied" ]
        }
    }
}

.. and the specific (custom) pattern itself:

SW_MS (?:[0-9]{1,4})
SW_DUR_MS (?:%{SW_MS:MS:int}%{SPACE}(ms))

SW_SEC (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
SW_MIN (?:[0-5]?[0-9])
SW_DUR_SEC (?:%{SW_SEC:SEC:int}%{SPACE}(sec))
SW_DUR_MIN (?:%{SW_MIN:MIN:int}%{SPACE}(min))
SW_DURATION (?:(\[Duration:%{SPACE}%{SW_DUR_MS}\])|(\[Duration:%{SPACE}%{SW_DUR_SEC}(,)%{SW_DUR_MS}\])|(\[Duration:%{SPACE}%{SW_DUR_MIN}(,)%{SW_DUR_MS}\])|(\[Duration:%{SPACE}%{SW_DUR_MIN}(,)%{SW_DUR_SEC}(,)%{SW_DUR_MS}\]))

I am running on Logstash v2.3.4, Elasticsearch v2.3.4, Kibana v4.5.3 and Filebeat v1.2.3 with only this one single server logging to my Elastic Stack. Logging levels are default not debug level.

How i can better find out why this started happening without any config changes?

2

What does the actual document stored in ES look like? How is that field mapped? Use ES's get mapping API.

3

Thank you for your response Magnus

Using the following query in sense GET filebeat-2016.07.26/_all/_mapping/field/MS,SEC,MIN i get the following output from the day before and after:

    "MS": {
      "full_name": "MS",
      "mapping": {
        "MS": {
          "type": "long"
        }
      }
    }

... and the following output from the day with the reported issue:

    "MS": {
      "full_name": "MS",
      "mapping": {
        "MS": {
          "type": "string",
          "index": "not_analyzed",
          "ignore_above": 1024
        }
      }
    }

An example of the document with this MS field in it is as follows:

{
  "_index": "filebeat-2016.07.27",
  "_type": "swAuditLog",
  "_id": "AVYr3M9K91aMV7UCrCTG",
  "_version": 1,
  "found": true,
  "_source": {
    "message": "[2016/07/27 12:17:36][AuditLog] [INFO]  [Duration: 113 ms]",
    "@version": "1",
    "@timestamp": "2016-07-27T10:17:36.000Z",
    "beat": {
      "hostname": "Swimmer",
      "name": "Swimmer"
    },
    "type": "swAuditLog",
    "offset": 50347146,
    "input_type": "log",
    "count": 1,
    "fields": null,
    "source": "c:\\Test\\Log\\test.log",
    "host": "Swimmer",
    "LoggedDateTime": "2016/07/27 12:17:36",
    "Date": "2016/07/27",
    "Time": "12:17:36",
    "Duration": "[Duration: 113 ms]",
    "MS": 113,
    "Bytes": 882
  }
}

Please let me know if i have missed any information you asked for.

4

Hmm. I'm not sure why, but on that problematic day the MS field was apparently mapped as a string. Without inspecting all documents with an MS field in the index it's impossible to tell why. I suggest you use an index template to explicitly assign mappings to your fields instead of relying on the automatic mapping to do the right thing.

5

That's fair enough, and thanks again i appreciate the help.
I will see about getting a predefined template in place to avoid this issue in the future.