Remove_field in csv filter section

Elastic Stack Logstash
1

Hello,

I'm trying to remove "extra" fileds while parsing csv files (separator is ; ).
My filter section is that one:

filter {
csv {
separator => ";"
source => message
columns => ["ID","Value","Timestamp","DateAction","ServerName"]
remove_field = ["message","host","path","@version","@timestamp"]
}
}

when remove_field is commented (#remove_field) it's working but some extra fields are added : message, host, path, @version, @timestamp
How car I remove them. When trying I have an error.

Regards

Driss

2

Those are default fields that (I think) Logstash adds. I think you (probably?) shouldn't remove them? I'd be interested in learning if this would actually break anything.

3

For something like csv's, removing message if the csv parses successfully makes sense, since otherwise you are indexing everything twice. There should be no issue with removing path if you do not care which file a document came from, nor host if you don't care where the file was. I cannot remember what elasticsearch does if the message is missing the @version or @timestamp fields. It may well add them in itself.

remove_field = ["message","host","path","@version","@timestamp"]

should be

remove_field => ["message","host","path","@version","@timestamp"]
4

Did you try removing the field in a new block after the csv block using mutate?

5

@arisbanach That would also work, what I meant was that the original user's problem was that they were missing the > in the remove_field directive in their configuration, which is why they got an error if they did not comment it out.

And I tested it, and elasticsearch does not object to documents missing @version and @timestamp. Obviously there is a lot of functionality that requires timestamps, but if Amouzigh_Driss's use case does not require that functionality I think he will be OK.

6

Thank you all for your answers.
The missing > was just when I wanted to post the issue.
I have already timestamp infirnation in csv files. Thus the additional field @timestamp has no added value.
Will try tomorrow and tell you.
Best regards
Driss

7

Yeah. It's working quit fine.Thanks

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.