Filebeat @metadata object, can we remove this object before publishing?

[sentient]

I'm using filebeat 7.1.1
with a simple log file input

filebeat.inputs:
- type: log

When the events are published, there is a "@metadata" object inserted in the event.

 { 
   "@timestamp": "2019-07-10T18:24:56.312Z",                                                        
   "@metadata": {                                                                                       
     "beat": "",                                                                                                                                       
     "type": "_doc",                                                                                                                                                           
     "version": ""                                                                                                                                               
   },     
   ...
   "agent": {                                                                                           
      "type": "filebeat", 
      "ephemeral_id": "97f29009-c620-4746-9598-66bb651b4707",
      "hostname": "pop-os", 
      "id": "38a65a07-1a0a-4e0a-a651-bc83065f73be",
      "version": "7.1.1"
  },
  ...

},

One observation is that the 'beat' and 'version' field are empty. But the data is already in the "agent" object.

How can I remove the "@metadata" object from the events before they are published to the output?

I tried

  processors:
     - drop_fields:
         fields: ["@metadata"]

without much success.

I also could not find the documentation on the "@metadata"
at https://www.elastic.co/guide/en/beats/filebeat/7.1/exported-fields-beat.html

[steffens (Steffen Siering)]

You can not remove it. Internally it is dropped by the Elasticsearch output. Normally it is used in conjunction with Logstash (e.g. filebeat -> kafka -> logstash -> elasticsearch) and should ship information about the index to use. It can actually be different from agent. But it should not be empty, though.

[sentient]

Thanks for the explanation. I never used LogStash. Would be nice to have this once removed and save some bytes that have to go over the network

[martinr_ubi (Martin R.)]

Maybe there is a misunderstanding here.
What will be the output that you will really use in production?
You say save bytes over the wire but your question is about the local file output of filebeat. Which means your outputting to the local filesystem. No wire here.

You dont use logstash either, so were you inspecting the events with the file output but in reality you’ll use the elasticsearch output?

If you ship to ES with the ES output, those fields are removed and not sent to ES. So the bytes are saved if you use the ES output. And because those fields are for logstash they need to be sent when the logstash output is used.

If you use the file output but then send the events over the wire it means you’re configuring filebeat to write to files on disk and then using something else to read them and finally send them over the wire? So that other soft could remove the meta field before shipping the events then.

I’m curious to understand your setup in the context of “over the wire” while you report using the file output... What am I missing?

[sentient]

I'm just looking at the journalbeat logs from filebeat. It 'suggested' that those event values were being send to the output. Glad to hear that this is not the case.

[meister]

How about sending to Kafka?

I want to drop those metadata and just send the message part.

My usecase is:
Read logs via filebeat
Send logs to kafka

Kafka is managed by other team with configurations we can't touched.
They can't parse our data because of unnecessary metadata.

We'd like to avoid using Logstash for this.