Copying field values

Elastic Stack Beats
1

I want to copy the value of the host.hostname field to host.name or otherwise ensure that host.name takes this value.

I'm collecting syslog and auth data from a number of hosts on a single machine and shipping to Elastic Search from there via filebeat and the system module. The only issue is that visualizations and apps (e.g. SIEM app) default to looking at the host.name field for host information. The actual host from which data originates is in the files being ingested, and ends up in host.hostname.

I've tried using a couple of processors (copy-field, rename), but am not having much luck and getting very frustrated. Any guidance would be greatly appreciated.

2

OK! Some progress! I can overwrite the value in the host.name field with a value from another field using processors... I just can't figure out how to access the host.hostname field I guess?

As a silly test, this works:

processors:
    - drop_fields:
        fields: ["host.name"]
    - copy_fields:
        fields:
            - from: input.type
              to: host.name
        fail_on_error: false
        ignore_missing: true

This does not:

processors:
    - drop_fields:
        fields: ["host.name"]
    - copy_fields:
        fields:
            - from: host.hostname
              to: host.name
        fail_on_error: false
        ignore_missing: true

Is that the sort of problem anyone can help me to solve? Is it just a matter of figuring out how to name the field at this stage of the game?
I see host.hostname in the JSON in the Discover tab, but maybe I need to call it something else in my filebeat.yml?

3

OK, I just cannot figure out how to access the host.hostname field from these processors. Is it because the field is exported via the system module?

4

Did you try to rename the field: https://www.elastic.co/guide/en/beats/filebeat/master/rename-fields.html ?

5

The issue seems to be that I cannot access the host.hostname field via processors. I have tried drop, copy_field, and rename. I can access/change most other fields, and I can see host.hostname in the JSON for documents associated with the index:

...
"host": {
  "hostname": "NAME_I_CARE_ABOUT",
  "name": "NAME_I_DONT"
},
...

I've tried referring to both host.hostname and the alias system.syslog.hostname.

6

Can you try with add_host_metadata at the beginning of the processors section?

7

Sorry for the slow response. I was away from work for a couple of days without reliable access.

Adding the add_host_metadata processor brings a lot of additional information about the device shipping data into my documents/JSON, but does not make the host.hostname field available to rename.

Processors now appear to read host.hostname as containing the value in the host.name field, which is not the case when I look at the JSON.

The behaviour of this processor is very unexpected:

processors:
    - add_host_metadata: ~
    - rename:
        fields:
          - from: host.hostname
            to: new_name
        fail_on_error: false
        ignore_missing: true

A new_name field will be created, containing the value of host.name, and both host.hostname and host.name will remain as document fields... which is to say, no rename operation takes place at all. I'm fairly stumped.

8

I feel like I'm basically trying to do exactly what @madduck proposed here:

That gives me some hope I'm not totally out to lunch, but I'm having a heck of a time getting my hooks into the field I'm after.

9

I feel like I have a working solution. It's not perfect, but adding a processor to the ingest pipeline accomplishes the task I need.
I added a set processor to the ingest pipelines for the auth and syslog data, and am now able to interact with the SIEM app sensibly and see all my hosts.

{
  "set" : {
    "field" : "host.name",
    "value" : "{{host.hostname}}"
  }
},

For anyone finding this thread and needing to accomplish a similar task, consider editing either the default pipelines (JSON files in /user/share/filebeat/MODULE_NAME/SUB_MODULE/ingest/pipeline.json) or updating exiting pipelines (https://www.elastic.co/guide/en/elasticsearch/reference/current/put-pipeline-api.html), depending upon your set up.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.