Logstash does not forget the previous csv header coming from filebeat

Elastic Stack Logstash
1

Hi,
My issue:

  • When I pass csv files from filebeat with different header, csv output does not match the right column with the right value.
    Does someone know to manage that, forget the previous csv header read by log stash and coming from filebeat ?
    Example :
    file1.csv :
    c1, c2, c3
    1, 2, 3
    output: c1=1, c2=2, c3=3
    file2.csv:
    c1, c2, c3, c4
    1,2,3,4
    output: c2=1, c3=2, c4=3

I set up my logstash config as below:
indent preformatted text by 4 spaces

input{
beats{
port => "5044"
}
filter{
if ([fields][log_type]=="bucking"){
csv{
separator => ","
autodetect_column_names => true
autogenerate_column_names => true
skip_header => false
skip_empty_columns => false
skip_empty_rows => false
}
mutate{
convert => {
"Output_id" => "integer"
"Diameter" => "float"
"Price" => "float"
"TotalValue" => "float"
"Volume" => "float"
"NominalVolume" => "float"
"RealVolume" => "float"
"SawdustVolume" => "float"
"NbSol" => "integer"
"NumShapePLC" => "integer"
"TimeDisp" => "float"
"TimeOpti" =>"float"
"TimeWait" => "float"
"TimeSend" => "float"
"TimeOptiMin" =>"float"
"TimeOptiMax" => "float"
"TimeOptiAverage" => "float"
"TimeOptiTotal" => "float"
"EtatSolution" => "integer"
"ValeurReelle" => "float"
"Version_TVL" => "string"
}
}
mutate {
copy => {
"[fields][log_type]" => "bucking"
}
}
prune{
whitelist_names => ["Output_id","NumShapePLC","Version_TVL"]
}
}
}
output{
elasticsearch{
hosts => "localhost:9200"
index =>"%{[fields][log_type]}"}
stdout{}
}

2

If you are using a csv filter, especially if you are using autodetect_column_names, then every event has to have the same fields.

You could use a different csv filter for each file, and route the events through the csv filters based on [log][file][path].

3

The thing is that I have already multiple filter.
first csv filter for : [fields][log_type1]_1.csv, [fields][log_type1]_2.csv, [fields][log_type1]_3.csv
second csv filter for: [fields][log_type2]_1.csv, [fields][log_type2]_2.csv, [fields][log_type2]_3.csv
I have 7 csv filter.

But for the same log type, sometime a new field appear.
It is working well when i restart logstash between 2 csv files with same [fields][log_type], the csv header is recreated.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.