Filebeat to logstash using csv filter not working

[nareshreddyp (Naresh Reddy)]

Hello,

I have a csv file with following entries (Each column is already with double quotes)
"2019/11/07 13:19:18.538";"138.106.61.153";"sxisl";"sxist0";"mxobjecttype:(""BUSINESSOBJECT"")";"xx";"0";"2";"1";"1";"118820";

When filebeat sends to logstash it is sending with ", so the csv filter is not working. How to filter the data coming from csv file using filebeat _

Logstash ruby output

My file beat config

filebeat.inputs:

• input_type: log
  paths:
  • /data/3dspaceindex/cv/run/searchserver-ss0/search-reporting/*
    tags: ["otb,3dspaceindex"]
    output.logstash:
    hosts: ["localhost:5065"]

logstash config

input {
beats {
port => 5065
}
}

filter {
if "3dspaceindex" in [tags]
{
mutate {
gsub => ["message", """, ""]
}
csv {
skip_empty_columns => "true"
skip_header => "true"
separator => ";"
columns => [ ""#timeStamp","apiclient_ip","query_logic","query_target","query_querystring","query_language","query_start","query_hf","answer_nmatches","answer_nhits","time_total","query_full","query_id","query_origin","answer_status"" ]
}
date
{
match => ["#timeStamp", "yyyy/MM/dd HH:mm:ss.SSS"]
remove_field => [ "timestamp" ]
remove_field => [ "message" ]
}
mutate {
convert => {
"query_start" => "integer"
"query_hf" => "integer"
"answer_nmatches" => "integer"
"answer_nhits" => "integer"
"time_total" => "integer"
"query_id" => "integer"
}
}
if [message] =~ /^#timestamp/ {
drop { }
}
}

}

output {
stdout { codec => rubydebug }
}

[Badger]

@nareshreddyp

mutate {
gsub => ["message", """, ""]
}

I think you should remove this.

[nareshreddyp (Naresh Reddy)]

Hi,

I have tried removing that but the result was always with '"'
logstash is not filtering correctly.

Regards,
Naresh