Trying to avoid manual changes to files pre-logstash

I have csv files that include bad output from SQL output that logstash fails to parse because of quote marks that stay open. Example line:

I am fixing them manually with sed before running logstash.
sed -i 's/"N;/"N";/g' *.csv
sed -i 's/\"N$/"N"/g' *.csv

Using logstash conf only, how can I run these sed commands or change the file using these sed patterns before logstash tries to ingest them?
I saw the "exec" plugin, but need to run those before logstash parses new data, not at a set interval.

Thanks in advance!

You can use a mutate filter (its gsub option) to process the input line before the csv filter.

Thank you, I would really appreciate an understanding of how to escape characters using this method.
Right now I fail. Even when trying to set this in logstash.yml:

config.support_escapes: true

Using double quotes as an example, how do I escape " inside a conf file gsub clause?
Thanks !

You can use the double quotes without escapes if you make the string itself single-quoted.

1 Like

OK that will do it for most of the bad line, but I have to escape the end of line because of the last "N
So the equivalent of using $ in sed is also needed for the conf file. Any chance you can think of a way to escape it or an alternative?

Thanks !

Why would there be a need to escape the dollar sign?

I don't need to escape the dollar sign, I need to escape the "end of line" symbol. In sed that symbol is a dollar sign

Yes, I know the meaning of a dollar sign in a regular expression. And if you want the dollar sign to mean "end of line" why would you want to escape it?

Now I understand the question, sorry. I need to replace end-of-line with a double quote inside mutate->gsub

Instead of "N (-->end of line)
I need to create "N"

Inside gsub

Just use the same expression as with sed apart from your (unnecessary) escaping of the double quote, i.e. "N$.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.