Parse cvs file with a key and multiple values

[Steven_Lin]

Hi!
i have a cvs file with a key and multiple value by date to show the devices status like this:
deviceid, 2018-04-01, 2018-04-02, 2018-04-03, ...
d001, 0, 0, 1, ...
d002, 1, 1, 1, ...
d003, 0, 1, 1, ...
...

can't find the way to deal with multiple values issue for logstash.
anyone got the solution?

Regards, Steven

[NerdSec (Nachiket)]

Hi Steven,

Using a csv filter you can directly parse a csv. Why is there a need for using key-value logic?

https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html

[Steven_Lin]

Hi! NerdSec,
Thanks for your reply to my question. i think i'm new to logstash and trying to find any helpful information to deal with the need of translating csv file data to elasticsearch. Anyway, i think i need to import and transform csv below:

deviceid, 2018-04-01, 2018-04-02, 2018-04-03, ...
d001, 0, 0, 1, ...
d002, 1, 1, 1, ...
d003, 0, 1, 1, ...
...

to elasticsearch of following records:

recorddate deviceid status
2018-04-01 d001 0
2018-04-02 d001 0
2018-04-03 d001 1
...
2018-04-01 d002 1
2018-04-02 d002 1
2018-04-03 d002 1
...

then i can see the statistic chart from Kibana for the devices availability.
Don't know how the logstash configuration file to do to this kind of transformation (the columns are not fixed, according to the number of date).
Or maybe i miss something or misunderstand the usage of it. Any information will be appreciated, thanks.

Regards, Steven

[NerdSec (Nachiket)]

Hi Steven,

This does not seem to be possible with any of the existing filters. You might have to either transpose the csv before processing or write a ruby code to do the same.

I am not very good at the ruby filter, maybe someone else could help.

https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html