Date parsing problem

Elastic Stack Logstash
1

I don't know why, but the date filter is not parsing the date field and I don't see any error logs regarding it. The docs do have the _dateparsefailure tag though. My config:

  xml {
    source => "message"
    xpath => ["/element/date/text()", "date"]
    store_xml => false
  }
  date {
    match => ["date", "yyyyMMdd"]
    target => "date"
  }

The date fields are all like 20180425. Any ideas why this is happened?

2

@arisbanach if you run it through with the following output, what do you see?

output { stdout { codec => rubydebug } }
4

@Mike.Barretta The output shows the same as what I see in Kibana. There isn't anything extra that I can see that explains why it fails to parse.

5

Date is an array. Try this (force_array => false on the xml filter does not stop path making everything an array).

  date { match => [ "date[0]", "yyyyMMdd"] target => "date" }
6

@Badger You're right! However, when I set force_array => false in the xml filter, it still outputs date as an array for some reason.

7

Actually, this might be related to the xpath value I'm using to pull it. This has been happening with other fields as well, so I'm guess I'm just not understanding something about xpaths. Is there a way that I can concatenate multiple matches so that the output isn't stored in an array? Or will I need to use a different filter afterward to do that?

8

force_array => false works for store_xml, but not for xpath. If you look at the source it is not even referenced in the if @xpath block, just in the if @store_xml block.

9

So would you say a good way to handle this is just use another filter after the xml filter to merge any fields that are arrays into non-arrays first?

10

Yes. I tend to do this for each field. It is ugly but effective.

if [fieldx] { mutate { replace => { "fieldx" => "%{[fieldx][0]}" } } }
11

Thanks! You're amazing! How long have you been doing this for, if you don't mind me asking? I don't know if there is a normal learning curve with Logstash or if I'm just slow.

12

Thanks for jumping in and help out!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.