How to remove certain Filebeat output fields

Elastic Stack Beats
1

Hello,
I started to play with filebeat, just set it up on my local box. I have a newbie question.
I set the output to be local file right now, eventually i would like to set it to kafka.
While I examined the output from filebeat, by default, it outputs many fields we are not interested, e.g., @timestamp, @type, or @input_type. The only things we are interested is "hostname, message, and source". Those extra fields wasted space/bandwidth from log output.
so I am wondering how can i change the default configuration to remove those fields?
is filebeat.template.json the right place to change this? I tried to remove certain filebeat.template.json, it did not work., Is there a documentation link for this?
I am also curious how ppl use these fields? e.g., read_timestamp, i.e., @timestamp just in case it might be useful and I should keep them..

Thanks!
yan

2

Hi @filebeater,

People use logs in many different ways, here you have a very good example on how that many fields can help: https://www.youtube.com/watch?v=SuDQ3-FihQk. For instance, @timestamp allows you to narrow your analysis to an specific time range.

Elasticsearch is good at storing these values and optimize disk usage for them, so in general you should not care that much, but it all depends on your use case.

You can always drop fields you are not interested on by using drop_fields processor, check the doc for it https://www.elastic.co/guide/en/beats/filebeat/5.4/drop-fields.html. Anyway I think it won't help with the @ prefixed fields, as they are always added by filebeat

Welcome to the forum!

1 Like
3

@exekias thanks for the pointer!
@timestamp is not the real timestamp when the log entry is generated, but the timestamp when Filebeat process that log line. Most log libraries (e.g., log4j and glog) already log the timestamp, which is why i feel @timestamp is redundant and confusing.

Just to make sure I understand this, @timestamp can not be removed for now, I am wondering How can I submit the feature request to be able to remove @ prefixed fields timestamp?
Actually in one setup, we would like to remove all meta data fields, keep only original log message, i.e., the value of the "message" field.

thanks!
yan

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.