Dateparsefailure on conversion


(Pjanzen) #1

Hi All.

I have the following timestamp 20180612 21:07:10.197 and I am using the this match:

date {
      # 2018-06-04T07:25:13.943Z
      # 20180605 16:37:34.966
      match => ['ts', 'yyyymmdd hh:mm:ss.SSS']
      timezone => "CET"
      target => "@timestamp"
    }

I do however end up with a _dateparsefailure

An example logline is this:

20180610 16:49:02.647 core id=x2p11x00736y3Au012p1mC wf=smtp::6:39 msg="logid=data_accept_default ip=62.144.109.144 revdns=mx109e144.fagms.de canrelay=?? action=P6_ACCEPT_DEFAULT"

and the logstash part I use to decompose this line is as follows:

if [message] =~ /action=P/ {
    dissect {
      mapping => {
        "message" => "%{ts} %{+ts} %{message}"
      }
    }
    kv {
      source => "message"
        prefix => "imp_"
      }
    }
}

produces the following output:

   {
  "_index": "clog-2018.06.12",
  "_type": "doc",
  "_id": "jBZz9WMBP1gu3W4L3uma",
  "_version": 1,
  "_score": null,
  "_source": {
    "offset": 1985746470,
    "message": "core id=x2p11x00736y3Au012p1mC wf=smtp::6:39 msg=logid=data_accept_default ip=62.144.109.144 revdns=mx109e144.fagms.de canrelay=?? action=P6_ACCEPT_DEFAULT",
    "beat": {
      "version": "6.2.4",
      "hostname": "mx7.12345.net",
      "name": "mx7.12345.net"
    },
    "imp_wf": "smtp::6:39",
    "ts": "20180610 16:49:02.647",
    "imp_msg": "logid=data_accept_default",
    "imp_ip": "62.144.109.144",
    "@version": "1",
    "imp_action": "P6_ACCEPT_DEFAULT",
    "imp_revdns": "mx109e144.fagms.de",
    "source": "/var/log/bizimp/filters.log",
    "source_affiliate": "ukmail",
    "host": "mx7.12345.net",
    "@timestamp": "2018-06-12T19:22:47.311Z",
    "tags": [
      "CM",
      "beats_input_codec_plain_applied",
      "_dateparsefailure"
    ],
    "imp_id": "x2p11x00736y3Au012p1mC",
    "prospector": {
      "type": "log"
    },
    "imp_canrelay": "??",
    "geoip": {
      "continent_code": "EU",
      "country_code3": "DE",
      "location": {
        "lat": 51.2993,
        "lon": 9.491
      },
      "country_code2": "DE",
      "country_name": "Germany",
      "longitude": 9.491,
      "ip": "62.144.109.144",
      "latitude": 51.2993
    }
  },
  "fields": {
    "@timestamp": [
      "2018-06-12T19:22:47.311Z"
    ]
  },
"highlight": {
    "imp_action": [
      "@kibana-highlighted-field@P6_ACCEPT_DEFAULT@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1528831367311
  ]
}

I do not understand why this is happening...


(Colton) #2

Try replacing the lowercase mm in yyyymmdd to yyyyMMdd. Same thing with the hh for hours. Replace the lowercase hh's to HH.

See this link about dates:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html

I think that Logstash may be trying to parse the month as minutes since you are using lowercase m's. If that isn't the correct problem you should still probably fix it.


(Pjanzen) #3

Thanks, that did the trick...


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.