Is it possible to add to @metadata in Filebeat?

[Alex_Nelson]

I have the following

  processors:
    - add_fields:
        target: '@metadata'
        fields:
          index_prefix: 'flow_metadata_event'

which gives me the following using output.console (note there are 2 object of the same name, huh?)

{
  "@timestamp": "2021-01-28T15:54:24.000Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.7.1",
   "_id": "574e1ea7-76f9-4830-9ef0-189e2c9f115e"
  },

  "event_time": "2021-01-28T15:54:24Z",
  "@metadata": {
    "index_prefix": "flow_metadata_event"
  },
  .....
  "user": "autoperf",
}

And if I try to refer to %{[@metadata.index_prefix]} it returns null.
Note this approach works if I use something other than @metadata.
If I use top-level JSON or some sub-object like "meta", the elasticsearch output works as expected.
But I don't want my output cluttered with pointless data like the index-prefix.
It seems silly to write an Elasticsearch ingest processor just to remove it also.

Am I missing something obvious?

[aaron-nimocks]

• Realized you weren't using Logstash but leaving it here as an option.

I didn't test this but I believe if you change the index in your LS output that field is now accessible using %{[@metadata][beat] in LS.

output.logstash:
  hosts: ["localhost:5044"]
  index: flow_metadata_event

[Alex_Nelson]

Thank Aaron for your reply.

I just finished removing Logstash from out Elastic stack because we cannot tolerate the large resource requirements imposed on us by Logstash. Using Logstash is therefore not an option for me.

Are you saying that my use case is not (yet?) supported in Filebeat?

[aaron-nimocks]

Well to answer your question I don't think it's possible to add to @metadata they way you are trying.

Can you help me understand what you are trying to achieve? Typically beats will go into a filebeat-xxxxxxx index and you want to do something custom.

If you don't have any processing, templates, pipelines running on ES they why isn't setting index to flow_metadata_event in your ES output in beats config? Trying to understand why you are attempting to do this.

[Alex_Nelson]

I am trying to generate a filebeat.yml from a script.
The script is told which kinds of logs to ingest.
For example, one ingest may be /var/log/flow.log and I want the target elasticsearch index to be flow-2021.01.18. Another ingest may be /var/log/storage.log, and I want the target elasticsearch index to be storage-2021.01.18.

In my filebeat template I have some shared code to drop fields that filebeat adds that I don't want.
I also have a single output for all the ingest inputs.

output.elasticsearch:
    hosts: http://localhost:9200
    index: "%{[meta.index_prefix]}-%{+YYYY.MM.dd}"

The idea is to have the script load various requested log ingestions into the final rendered filebeat.yml. Each ingestion knows which index should be used and provides that index in @metadata so that the information does not show up in elasticsearch, but the correct index for each of the various inputs is used.

Right now, I have it working using "meta", but not "@metadata".

[aaron-nimocks]

Got it. thanks for explaining it in detail.

I believe you are doing it the best possible way since you aren't able to change the @metadata.

[Alex_Nelson]

I figured out a workaround -- use javascript instead of the add_fields:

processors:

- script:
      lang: javascript
      source: >
        function process(event) {
          event.Put("@metadata.index_prefix", "flow-metadata-events");
        }