How can I add add an extra field to all documents indexed by beats

Elastic Stack Beats
1

Hi all.

Lets imagine that I have a single elasticsearch cluster to store document from two different companies

Company1 has
server1
server2 and
server3

Company2 has
server1
server2 and
server3

For me the easier way to make the different between both 'server1's of Company1 and 2 is by creating and adding a new field called 'company' that for the filebeat installed in Company1 servers will have a value of 'Company1', same with fbs in Company2 servers but with a value of 'Company2'.

I've tried it with this setting for my FB

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: false

  paths:
    - /var/log/*.log

  fields:
    company: "company1"

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

I omited the conectivity's sections to ES and Ki as they work well.

I've found this URL
Filter and enhance data with processors

But I dont see how to set a value
Can someone pls explain to me with a basic example how can I achieve this ?

Thank you in advance and regards.

Carlos T.

2

Hello,

It is not clear what is the issue, what you did already add a custom field.

Having this in your filebeat.yml should work and add a field named fields.company

  fields:
    company: "company1"

Is it not working? It is not clear.

3

Hi Leandro.

Thanks for your help. This is exactly what's happening. I see documents indexed from this server where I've applied this filebeat.yml I pasted but in any of the documents the field 'company' has been created. So it is not working.

Thank you again and regards.

4

Just noted something in your configuration.

You have enabled set to false, is this right? You need to set it to true or filebeat will not collect any logs.

It is not clear how you have any documents in your Elastic if you have enabled set to false in your filebeat input. Do you have any other inputs?

1 Like
5

You are completely right sir, thank you very much for the advice. I had documents in my ES because I activated both 'system' and 'apache' modules and I was indexing other logs rather than those inside /var/log/*

Now I'm indexing documents from more logs and from Kibana -> Discover I can see a new field called 'field.company' with the expected values.

My question now is. Will this config in the filebeat apply to to the logs are are collected by suing the 'system' and 'apache' modules?

For what I've seen. It seems so.

Thanks again Leandro

6

I don't think so, this configuration will apply only to the input in which it is configured.

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: false

  paths:
    - /var/log/*.log

  fields:
    company: "company1"

If you want it to apply to everything this filebeat collect, I think you need to use the add_fields under the processors and remove the fields configuration from the filestream input.

You would need to change your filebeat.yml to something like this:

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: false

  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - add_fields:
      fields:
        company: "company1"
2 Likes
7

It is exactly as you said.

I've been comparing by querying with and without field.company and I've got exactly the same number of results.

Thank you Leandro. I really appreciate your time and help. Have a nice rest of the day D

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.