Convert multiline csv data to a single line csv in filter

Elastic Stack Logstash
1

Hi All

I have a csv file, which gives me response in multiline csv, i need convert that in a single line csv,

Here is the data

"header1","header2","header3"
"col1row1","col2row1","col3row1"
"col1row2","col2row2","col3row2"

if i use this below configuration, i get the response like this

{"message":"\"col1row2\",\"col2row2\",\"col3row2\"\r"}
{"message":"\"col1row1\",\"col2row1\",\"col3row1\"\r"}

my expected result it to get all the value of key "message" combined into a single value.
You can say its a case of combining all multiline csv value to a single line

{"message": "col1row1","col2row1","col3row1"
"col1row2","col2row2","col3row2"}

OR

{"message": "col1row1","col2row1","col3row1"\n
"col1row2","col2row2","col3row2"}

This is my conf file

input{
	file {
	      path => "/csv_data.csv"
		    sincedb_path => "/sincedb.txt"
          type => "csv"
          start_position => "beginning"
      }
	}
filter {
mutate {
remove_field => ["host","@version","@timestamp","type","path"]
}

		}

	output {
	 file {
	   path => "/test.csv"

	 }
	}

Please let me know how we can combine all the value. I also tried checking aggregation filter, but no example usecase found to be a valid for this

Thanks

2

Use a multiline codec on the file input.

3

Hi @Badger , thanks for reply. yes this works well .
I was trying same in jdbc input, and this was not working there, can you please tell the reason. In both the case the its multiline

I am using this from one of your posts

codec => multiline { pattern => "^Spalanzani" negate => true what => "previous" auto_flush_interval => 2 }

4

The codec option can be set on a jdbc input, but it is ignored.

5

oh ! that means this cannot be solved. Do you know how we can solve this through aggregation filter or a ruby code?

6

Hi @Badger, can you provide any suggestion as how we can achieve this for jdbc source

7

If you want to combine rows from the DB then an aggregate filter might be appropriate.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.