Csv parsing through logstash

[Raj_Kumar (Raj)]

Hi All,

Iam trying to parse my csv file to elasticsearch through logstash

my config looks like this

input {
      file {
          path => "/home/raj/Uge 16 Spyware-malware total7.csv"
          type => "trend_micro_spyware"
          start_position => "beginning"
  }
}

filter {
   if [type] == "trend_micro_spyware" {
    csv {
        columns => ["Received", "Generated","Product Entity/Endpoint", "Product", "Spyware/Grayware", "Endpoint","Source Host", "User", "Result","Detections", "Channel"]
        skip_empty_columns => "true"
        skip_empty_rows => "true"
        skip_header => "true"
    }
}
}

but am getting it in kibana like this

which includes a separate document with csv headers

and when i see the message its like this

Please help me to figure out this issue.

Thanks,
Raj

[Badger]

Whilst the file may be called .csv, it appears to be either space or tab separated, so you need to supply the separator option to the csv filter.

[Raj_Kumar (Raj)]

Thank you Badger for the info , i tried to use the separator

    csv {
        columns => ["Received", "Generated","Product Entity/Endpoint", "Product", "Spyware/Grayware", "Endpoint","Source Host", "User", "Result","Detections", "Channel"]
        separator => " "
        skip_empty_columns => "true"
        skip_empty_rows => "true"
        skip_header => "true"
    }


![image|690x345](upload://v2pyDEvzpEIh3Z6OU7EyvxT6eti.png)

1.First thing it splitted even the values, for example in the screenshot its one value like 'host details'
since it has space it splitted in  to two different values like host and detail separately
2.  Secondly, header was not removed ,empty columns and rows was not removed

Thanks,
Raj

[Badger]

Check out the skip_header and skip_empty_rows options for the filter.

If your fields really are space separated and contain embedded spaces without quotes then the format is ambiguous and csv will not be able to parse it. However, the message looks like it is either tab separated (which csv can handle) or fixed-width (which grok would be better for).