How to remove certain fields from filebeat index

Elastic Stack Beats
1

Hi Team,

I am new to Elasticsearch and we are running a POC on Elasticsearch. We would like to remove few fields from the index documents which are not relevant. Please help us to remove this from newly creating index and existing index.

I looked at the kibana->management/elasticsearch/index_management/Templates and could see those fields , can i remove it from there so that its not created for new index.

Thanks,
Jijo John

1 Like
2

Hello John,
You should use the drop_fields processor.
https://www.elastic.co/guide/en/beats/filebeat/current/drop-fields.html
Example:

processors:
  - drop_fields:
      fields: ["host.name", "ecs.version", "agent.version", "agent.type", "agent.id", "agent.ephemeral_id", "agent.hostname", "input.type"]
#  - add_host_metadata: ~
#  - add_cloud_metadata: ~
1 Like
3

Thank you Very much Abhishek. We will try this and confirm soon.

4

Its working fine. thank you.

Can we apply similar settings at elasticsearch server instead of editing this at clients (beats)

5

Hi John,
Yes it is possible to apply this setting at ElasticSearch server.
Please refer
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html

FYI, disabling it at client level reduces the data that is transferred to ElasticSearch as well as puts less burden on ElasticSearch server to ignore fields.

Please consider the above point before coming to final decision.

Thanks
Abhishek

6

Thank you Abhishek. We are planning to have it disabled at client level only

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.