Incident ID,Status,Resolved By,Resolution Breached,Resolution Month,Resolution Date,Resolution Date & Time,
IM02370568,Closed,GUNTURS2,FALSE,1,1/4/2016,1/4/2016 10:35
IM02370648,Closed,PRASADR6,FALSE,1,1/1/2016,1/1/2016 22:51
i am trying to read a the above csv file
The heading(1st line) need to be read and pass it to filter section of csv plugin
The data also need to be read and pass it further for parsing.
i would like to pass the heading dynamically to csv plugin of logstash, instead of hardcoding the headings.
Could you please help me out in resolving this issue or at-least an alternative for this problem.
The CSV filter is stateless, meaning that it handles each event without knowing anything about any previous events. Also the config is parsed and loaded at LS start before any events have been processed.
There is no way at the moment to do this very dynamically.
How many different CSV structures do you need to parse?
Does the filename and/or path contain a clue to the type of CSV structure?
How many different CSV structures do you need to parse?
> There is no limit for CSV structures, at present we have done for 2 but in future we are expecting more, so would like to generalize for all the structures.
Does the filename and/or path contain a clue to the type of CSV structure?
> Yes, filepath can be in one location(we can hard code the path) but filename varies.
Only the very first line of the very first file will be the columns for all other files - because the CSV filter does not keep a map of file_path -> columns internally.
If you have to restart Logstash while it is half way though a file then the first line of that file will not be re-read and the columns will become some arbitrary set of values.
So if you have two different CSV structures, say structure-1 with columns "a","b" and structure-2 with "c","d"
then... Can you put all files with structure-1 into a sub-folder called structure-1 and so on?
If so then you can use regex if conditionals to separate the files so that they flow through their own csv filter. You still have to take into account the second pitfall of my previous post unless you hard code the columns for each structure.
Logstash do not have an automatic way of doing this, because by design we strive for statelessness and the file input (and Filebeat) is designed to resume reading from where reached the previous time.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.