Add (;) sépartor in csv file before analysing it

[ismael_boumedien]

Hi guys,

I want to add ( sépartor in csv file before analysing it.

Example befor e:
2017/01/08-18:32:08mydomain.fr;151;ACC_EFFETMETEO_20170108181410.tar.gz;/arc/dacc/staging/max/SCORE;1
Result wanted:
2017/01/08-18:32:08;mydomain.fr;151;ACC_EFFETMETEO_20170108181410.tar.gz;/user/arc/staging/max/SCORE;1

Can you please give a tip to do it in logstash.

Best regards

Ismael

[Christian_Dahlqvist]

Why not instead run a grok filter to capture the timestamp and put the rest of the log line into a separate variable and then apply the csv filter to this field?

[ismael_boumedien]

@Christian_Dahlqvist

to

Hi,

Is it authorized to grok before csv filter?

Best regards

Ismael

[Christian_Dahlqvist]

You can run processing filters in any order you want. It is not uncommon to parse part of the message using one filter and then apply another to some of the fields.

[ismael_boumedien]

Hi,

Thanks for your help. It's working.

Best regards

Ismael

--- For information
input {
file {
path => "/projects/elastic/logs/stream.log"
start_position => "beginning"
}
}
filter {
grok{
match => ["message", "(?\d{4}/\d{2}/\d{2}-\d{2}:\d{2}:\d{2})%{GREEDYDATA:serviceMessage}"]
}
csv {
source => "serviceMessage"
columns => ["host", "number", "filename", "path", "Status"]
separator => ";"
}
date {
locale => "en"
match => ["time", "YYYY/MM/dd-HH:mm:ss"]
timezone => "Europe/Paris"
add_tag => "checkpoint_fw"
}
mutate {
add_field => { "IndexType" => "hadoop_prod" }
}
if "_grokparsefailure" in [tags] {
drop {}
}
if "_csvparsefailure" in [tags] {
drop {}
}

}
output {
stdout { codec => rubydebug }
}