Dateparsefailure on conversion

[pjanzen (Paul Janzen)]

Hi All.

I have the following timestamp 20180612 21:07:10.197 and I am using the this match:

date {
      # 2018-06-04T07:25:13.943Z
      # 20180605 16:37:34.966
      match => ['ts', 'yyyymmdd hh:mm:ss.SSS']
      timezone => "CET"
      target => "@timestamp"
    }

I do however end up with a _dateparsefailure

An example logline is this:

20180610 16:49:02.647 core id=x2p11x00736y3Au012p1mC wf=smtp::6:39 msg="logid=data_accept_default ip=62.144.109.144 revdns=mx109e144.fagms.de canrelay=?? action=P6_ACCEPT_DEFAULT"

and the logstash part I use to decompose this line is as follows:

if [message] =~ /action=P/ {
    dissect {
      mapping => {
        "message" => "%{ts} %{+ts} %{message}"
      }
    }
    kv {
      source => "message"
        prefix => "imp_"
      }
    }
}

produces the following output:

   {
  "_index": "clog-2018.06.12",
  "_type": "doc",
  "_id": "jBZz9WMBP1gu3W4L3uma",
  "_version": 1,
  "_score": null,
  "_source": {
    "offset": 1985746470,
    "message": "core id=x2p11x00736y3Au012p1mC wf=smtp::6:39 msg=logid=data_accept_default ip=62.144.109.144 revdns=mx109e144.fagms.de canrelay=?? action=P6_ACCEPT_DEFAULT",
    "beat": {
      "version": "6.2.4",
      "hostname": "mx7.12345.net",
      "name": "mx7.12345.net"
    },
    "imp_wf": "smtp::6:39",
    "ts": "20180610 16:49:02.647",
    "imp_msg": "logid=data_accept_default",
    "imp_ip": "62.144.109.144",
    "@version": "1",
    "imp_action": "P6_ACCEPT_DEFAULT",
    "imp_revdns": "mx109e144.fagms.de",
    "source": "/var/log/bizimp/filters.log",
    "source_affiliate": "ukmail",
    "host": "mx7.12345.net",
    "@timestamp": "2018-06-12T19:22:47.311Z",
    "tags": [
      "CM",
      "beats_input_codec_plain_applied",
      "_dateparsefailure"
    ],
    "imp_id": "x2p11x00736y3Au012p1mC",
    "prospector": {
      "type": "log"
    },
    "imp_canrelay": "??",
    "geoip": {
      "continent_code": "EU",
      "country_code3": "DE",
      "location": {
        "lat": 51.2993,
        "lon": 9.491
      },
      "country_code2": "DE",
      "country_name": "Germany",
      "longitude": 9.491,
      "ip": "62.144.109.144",
      "latitude": 51.2993
    }
  },
  "fields": {
    "@timestamp": [
      "2018-06-12T19:22:47.311Z"
    ]
  },
"highlight": {
    "imp_action": [
      "@kibana-highlighted-field@P6_ACCEPT_DEFAULT@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1528831367311
  ]
}

I do not understand why this is happening...

[CDR (Colton)]

Try replacing the lowercase mm in yyyymmdd to yyyyMMdd. Same thing with the hh for hours. Replace the lowercase hh's to HH.

See this link about dates:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html

I think that Logstash may be trying to parse the month as minutes since you are using lowercase m's. If that isn't the correct problem you should still probably fix it.

[pjanzen (Paul Janzen)]

Thanks, that did the trick...