# Logstash does not forget the previous csv header coming from filebeat

**URL:** <https://discuss.elastic.co/t/logstash-does-not-forget-the-previous-csv-header-coming-from-filebeat/256910>\
**Category:** Logstash\
**Created:** [November 27, 2020, 6:50pm UTC](https://discuss.elastic.co/t/logstash-does-not-forget-the-previous-csv-header-coming-from-filebeat/256910 "2020-11-27T18:50:28Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jason.greffier](https://avatars.discourse-cdn.com/v4/letter/j/a88e57/32.png) [@jason.greffier](https://discuss.elastic.co/u/jason.greffier)\
**Post date:** [November 27, 2020, 6:50pm UTC](https://discuss.elastic.co/t/logstash-does-not-forget-the-previous-csv-header-coming-from-filebeat/256910/1 "2020-11-27T18:50:28Z")

</div>

Hi,  
My issue:

- When I pass csv files from filebeat with different header, csv output does not match the right column with the right value.  
Does someone know to manage that, forget the previous csv header read by log stash and coming from filebeat ?  
Example :  
file1.csv :  
c1, c2, c3  
1, 2, 3  
output: c1=1, c2=2, c3=3  
file2.csv:  
c1, c2, c3, c4  
1,2,3,4  
output: c2=1, c3=2, c4=3

I set up my logstash config as below:  
indent preformatted text by 4 spaces

input{  
beats{  
port =\> "5044"  
}  
filter{  
if ([fields][log\_type]=="bucking"){  
csv{  
separator =\> ","  
autodetect\_column\_names =\> true  
autogenerate\_column\_names =\> true  
skip\_header =\> false  
skip\_empty\_columns =\> false  
skip\_empty\_rows =\> false  
}  
mutate{  
convert =\> {  
"Output\_id" =\> "integer"  
"Diameter" =\> "float"  
"Price" =\> "float"  
"TotalValue" =\> "float"  
"Volume" =\> "float"   
"NominalVolume" =\> "float"  
"RealVolume" =\> "float"  
"SawdustVolume" =\> "float"  
"NbSol" =\> "integer"  
"NumShapePLC" =\> "integer"  
"TimeDisp" =\> "float"  
"TimeOpti" =\>"float"  
"TimeWait" =\> "float"  
"TimeSend" =\> "float"  
"TimeOptiMin" =\>"float"  
"TimeOptiMax" =\> "float"  
"TimeOptiAverage" =\> "float"  
"TimeOptiTotal" =\> "float"  
"EtatSolution" =\> "integer"  
"ValeurReelle" =\> "float"  
"Version\_TVL" =\> "string"   
}  
}  
mutate {  
copy =\> {  
"[fields][log\_type]" =\> "bucking"  
}  
}  
prune{  
whitelist\_names =\> ["Output\_id","NumShapePLC","Version\_TVL"]  
}  
}  
}  
output{  
elasticsearch{  
hosts =\> "localhost:9200"  
index =\>"%{[fields][log\_type]}"}  
stdout{}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 27, 2020, 8:30pm UTC](https://discuss.elastic.co/t/logstash-does-not-forget-the-previous-csv-header-coming-from-filebeat/256910/2 "2020-11-27T20:30:25Z")

</div>

If you are using a csv filter, especially if you are using autodetect\_column\_names, then every event has to have the same fields.

You could use a different csv filter for each file, and route the events through the csv filters based on [log][file][path].

---

<div class="post-metadata">

**Author:** ![jason.greffier](https://avatars.discourse-cdn.com/v4/letter/j/a88e57/32.png) [@jason.greffier](https://discuss.elastic.co/u/jason.greffier)\
**Post date:** [November 27, 2020, 8:36pm UTC](https://discuss.elastic.co/t/logstash-does-not-forget-the-previous-csv-header-coming-from-filebeat/256910/3 "2020-11-27T20:36:40Z")

</div>

The thing is that I have already multiple filter.  
first csv filter for : [fields][log\_type1]\_1.csv, [fields][log\_type1]\_2.csv, [fields][log\_type1]\_3.csv  
second csv filter for: [fields][log\_type2]\_1.csv, [fields][log\_type2]\_2.csv, [fields][log\_type2]\_3.csv  
I have 7 csv filter.

But for the same log type, sometime a new field appear.  
It is working well when i restart logstash between 2 csv files with same [fields][log\_type], the csv header is recreated.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 25, 2020, 8:36pm UTC](https://discuss.elastic.co/t/logstash-does-not-forget-the-previous-csv-header-coming-from-filebeat/256910/4 "2020-12-25T20:36:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.