CSV within CSV


(Khoa Nguyen) #1

My CSV records have fields that are CSV themselves (with different delimiter). For example:

a,b,c,d1|d2|d3,e,f1|f2,g,h

How do I write a logstash configuration file to handle this?

Thanks


(Mark Walkom) #2

https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-separator should help


(juergen) #3

I think the question was: can i set a second separator in one CSV Input ?
probably not,
maybe with a second kv filter.......
https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html
have somebody other suggestions???


(Magnus Bäck) #4

The csv filter doesn't support nested CSV structures (for good reasons). Just use two csv filters in series.


(Khoa Nguyen) #5

Thanks for all your inputs. Magnus, I can't figure out how to construct two CSV filters in series. Could you give me a hint?


(Magnus Bäck) #6

Set the source parameter of the second csv filter to the name of the field produced by the first filter. Actually, in your case you have two columns (the first and the third) so you need three csv filters. Something like this (replace "..." with the desired field names, obviously):

csv {
  columns => ["one", "two", "three"]
  separator => "|"
}
csv {
  columns => [...]
  source => "one"
}
csv {
  columns => [...]
  source => "three"
}

(Khoa Nguyen) #7

Thanks! Works great


(system) #8