Dear Logstash Community,
Need your help. I am facing an issue with Network Switch Traffic that comes in the form of CSV files. The problem is some of the fields contain commas in it e.g. a URL and the string isn't encapsulated within quotation marks.
For example URL = http://url.com/search.?aspx=feed1@x1,x2,x3
What happens is now my message is and columns are blown out and I get something like:
URL, Bytes, column 30, column 31 etc. Where column 30 and 31 is new columns added by the CSV filter.
Below is my logstash conf file:
input {
file {
path => "/Users/wtaylor/Downloads/logstash-2.2.2/bin/sce/*.csv"
type => "usage"
start_position => "beginning"
}
}
filter {
csv {
columns => [
"TIMESTAMP",
"DUMMY",
"SUBSCRIBER_ID",
"PACKAGE_ID",
"SERVICE_ID",
"PROTOCOL_ID",
"SKIPPED_SESSIONS",
"SERVER_IP",
"SERVER_PORT",
"ACCESS_STRING",
"INFO_STRING",
"CLIENT_IP",
"CLIENT_PORT",
"INITIATING_SIDE",
"REPORT_TIME",
"MILLISEC_DURATION",
"TIME_FRAME",
"SESSION_UPSTREAM_VOLUME",
"SESSION_DOWNSTREAM_VOLUME",
"SUBSCRIBER_COUNTER_ID",
"GLOBAL_COUNTER_ID",
"PACKAGE_COUNTER_ID",
"IP_PROTOCOL",
"PROTOCOL_SIGNATURE",
"ZONE_ID",
"FLAVOR_ID",
"FLOW_CLOSE_IP_TYPE",
"SERVERIPv6ADDRESS",
"CLIENTIPv6ADDRESS"
]
separator => ","
}
}
output {
stdout { codec => json_lines }
}
I ideally need to get length of the array if split by commas and join the extra commas one of the fields e.g. INFO_STRING
Does anyone know how to achieve this.
Wayne