Trying to avoid manual changes to files pre-logstash


(Dor Juravski) #1

Hello,
I have csv files that include bad output from SQL output that logstash fails to parse because of quote marks that stay open. Example line:
"1";"2";"3";"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N;"N

I am fixing them manually with sed before running logstash.
sed -i 's/"N;/"N";/g' *.csv
sed -i 's/\"N$/"N"/g' *.csv

Using logstash conf only, how can I run these sed commands or change the file using these sed patterns before logstash tries to ingest them?
I saw the "exec" plugin, but need to run those before logstash parses new data, not at a set interval.

Thanks in advance!
JD


(Magnus Bäck) #2

You can use a mutate filter (its gsub option) to process the input line before the csv filter.


(Dor Juravski) #3

Thank you, I would really appreciate an understanding of how to escape characters using this method.
Right now I fail. Even when trying to set this in logstash.yml:

config.support_escapes: true

Using double quotes as an example, how do I escape " inside a conf file gsub clause?
Thanks !


(Magnus Bäck) #4

You can use the double quotes without escapes if you make the string itself single-quoted.


(Dor Juravski) #5

OK that will do it for most of the bad line, but I have to escape the end of line because of the last "N
So the equivalent of using $ in sed is also needed for the conf file. Any chance you can think of a way to escape it or an alternative?

Thanks !
JD


(Magnus Bäck) #6

Why would there be a need to escape the dollar sign?


(Dor Juravski) #7

I don't need to escape the dollar sign, I need to escape the "end of line" symbol. In sed that symbol is a dollar sign


(Magnus Bäck) #8

Yes, I know the meaning of a dollar sign in a regular expression. And if you want the dollar sign to mean "end of line" why would you want to escape it?


(Dor Juravski) #9

Now I understand the question, sorry. I need to replace end-of-line with a double quote inside mutate->gsub

Instead of "N (-->end of line)
I need to create "N"

Inside gsub


(Magnus Bäck) #10

Just use the same expression as with sed apart from your (unnecessary) escaping of the double quote, i.e. "N$.


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.