Logstash Date Format Failure

Elastic Stack Logstash
1

Hi I'm using filebeat to send some session count data to Logstash. The file type is csv and the data looks like this:

...
"01/02/2022 00:00:27.837", "0.99961"
"01/02/2022 00:00:28.853", "0"
"01/02/2022 00:00:29.240", "1.005890494747181"
....

I wanted to parse this data into two columns named "date" and "count", then put each row as a document, so I could plot a timeline graph where the y-axis is "count" and x-axis is "date".
So I tried to format the parsed "date" column to a timefield, but ran into a _dateformatfailure.

Here's part of my logstash.conf file.

filter {
  if [fields][codex] == "web-session" {
     mutate {
       split => [ "message", "," ]
       add_field => {
         "date" => "%{[message][0]}"
         "count" => "%{[message][1]}"
        }
        remove_field => ["message"]
    }
    date {
      match => [ "date", "MM/dd/YYYY HH:mm:ss.SSS" ]
      timezone => "Asia/Seoul"
      target => "convert_date"
      remove_field => [ "@timestamp"] 
    }

I used stdout to see how the parsed date field actually looked like, and found that it included quotation marks.

"date" => "\"01/02/2022 00:00:27.837"\"

I solved the problem by adding a gsub to delete the extra quotation marks, but my question is why did the date field contain extra quotation marks in the first place?

2

That is the solution I would have used.

Your message field "01/02/2022 00:00:27.837", "0.99961" is split into an array of two strings [ "\"01/02/2022 00:00:27.837\"", " \"0.99961\"" ]. Nothing in the filter is going to trim quotes or spaces unless you tell it to, and mutate+gsub is the way to do that here.

If you used a csv filter then that would remove quotes around a field but I suspect the space after the comma will result in an invalid quoting exception -- the entire field including the leading space would have to be quoted.

1 Like
3

@Badger
Thanks for the reply.
So I guess the reason was that the original csv file data already contained quotation marks which is unusual. I'm not very familiar with the csv file type so I reckoned having "" was normal. Thanks alot!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.