_csvparsing failure in logstash with delimeters

Elastic Stack Logstash
1

Hi all,
i am testing a logstash pipelines with the csv data which have approx 20 headers but i am trying with the 3 fields and csv delimeter tab is giving me parsing failures . i also tried with the quote_char but it is the same .
sample data events

(PDH-CSV 4.0) (Malay Peninsula Standard Time)(-480)	\\W01BUAMSDB3A\Processor Information(_Total)\% Processor Time	\\W01BUAMSDB3A\Memory\Available MBytes
01/31/2024 10:30:43.065	26.37411962	30405
01/31/2024 10:31:43.057	6.410672288	30360

this is my logstash configurations

filter {
        csv {
                separator => "\t"
                columns => ["timestamp", "processor_time", "available_memory" ]
                skip_empty_columns => "true"
                quote_char => " \\ "
        }
        ###  Date conversion ####
        date {
                match => [ "timestamp", "MM/dd/yyyy HH:mm:ss.SSS" ]
                target => "@timestamp_log"
                }
        date {
                match => [ "timestamp", "YYYY-MM-DD'T'HH:mm:ss.SSSSSSSSS'Z'" ]
                target => "@timestamp_raw"
        }
}
output
{
        stdout { codec => rubydebug }

i am getting the _csvparsing failures. pl suggest if any other options can be used to parse these colums.

      "event" => {
        "original" => "(PDH-CSV 4.0) (Malay Peninsula Standard Time)(-480)\t\\\\W01BUAMSDB3A\\Processor Information(_Total)\\% Processor Time\t\\\\W01BUAMSDB3A\\Memory\\Available MBytes"
    },
    "@timestamp" => 2024-02-05T03:30:33.874316203Z,
          "tags" => [
        [0] "_csvparsefailure"
    ],
2

According to the documentation you need a literal tab there, not \t.

3

sorry @Badger , i did not understand what is means by literal tab .
is it refereing to '\t' ?
i tried with it and its reporting the same response.
_csvparsefailures.

i tried with the "\tab" but its also same reporting.

"message" => "(PDH-CSV 4.0) (Malay Peninsula Standard Time)(-480)\t\\\\W01BUAMSDB3A\\Processor Information(_Total)\\% Processor Time\t\\\\W01BUAMSDB3A\\Memory\\Available MBytes",
          "host" => {
        "name" => "v097a.uat.abcd.com"
    },
          "tags" => [
        [0] "_csvparsefailure"
4

You cannot use \t, you cannot use \tab, you need to use the ASCII tab character, which is 0x09.

On my system I would have to reconfigure my editor not to replace tabs with spaces before trying to type a tab in a file.

5

i am in unix machine, and i tried with this format but i am getting this output even i tried to pass with one column in csv formats.

      "message" => "(PDH-CSV 4.0) (Malay Peninsula Standard Time)(-480)\t\\\\W01BUABCDF8B\\Processor
    "@timestamp" => 2024-02-06T02:40:00.603411786Z,
          "host" => {
        "name" => "x977a.vsi.abcd.com"
    },
     "timestamp" => "(PDH-CSV 4.0) (Malay Peninsula Standard Time)(-480)\t\\\\W01BUABCDF8B\\Processor
}

Below is my logstash configurations filter

filter {
 22         csv {
 23                 separator => ","
 24                 columns => [ "timestamp", "processor_time" ]
 25                 skip_empty_columns => "true"
 26                 #quote_char => ' \\" '
 27         }

may you suggest any other methods to parse these fileds ,

6

WIth this configuration

input { generator { count => 1 lines => [ '(PDH-CSV 4.0) (Malay Peninsula Standard Time)(-480)      \\W01BUABCDF8B\Processor' ] } }

output { stdout { codec => rubydebug { metadata => false } } }
filter {
    csv { separator => "    " columns => [ "timestamp", "processor_time" ] skip_empty_columns => "true" }
}

I get

     "timestamp" => "(PDH-CSV 4.0) (Malay Peninsula Standard Time)(-480)",
"processor_time" => "\\\\W01BUABCDF8B\\Processor",
       "message" => "(PDH-CSV 4.0) (Malay Peninsula Standard Time)(-480)\t\\\\W01BUABCDF8B\\Processor"

That is with a tab character in the message and in the separator option.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.