Hello,
I have csv files that include bad output from SQL output that logstash fails to parse because of quote marks that stay open. Example line: "1";"2";"3";"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N
I am fixing them manually with sed before running logstash. sed -i 's/"N;/"N";/g' *.csv sed -i 's/\"N$/"N"/g' *.csv
Using logstash conf only, how can I run these sed commands or change the file using these sed patterns before logstash tries to ingest them?
I saw the "exec" plugin, but need to run those before logstash parses new data, not at a set interval.
Thank you, I would really appreciate an understanding of how to escape characters using this method.
Right now I fail. Even when trying to set this in logstash.yml:
config.support_escapes: true
Using double quotes as an example, how do I escape " inside a conf file gsub clause?
Thanks !
OK that will do it for most of the bad line, but I have to escape the end of line because of the last "N
So the equivalent of using $ in sed is also needed for the conf file. Any chance you can think of a way to escape it or an alternative?
Yes, I know the meaning of a dollar sign in a regular expression. And if you want the dollar sign to mean "end of line" why would you want to escape it?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.