# Using a processor in a filebeat module may or may not actually find fields

**URL:** <https://discuss.elastic.co/t/using-a-processor-in-a-filebeat-module-may-or-may-not-actually-find-fields/191461>\
**Category:** Beats\
**Created:** [July 20, 2019, 12:07am UTC](https://discuss.elastic.co/t/using-a-processor-in-a-filebeat-module-may-or-may-not-actually-find-fields/191461 "2019-07-20T00:07:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![DPattee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dpattee/32/37772_2.png) [@DPattee](https://discuss.elastic.co/u/DPattee)\
**Post date:** [July 20, 2019, 12:07am UTC](https://discuss.elastic.co/t/using-a-processor-in-a-filebeat-module-may-or-may-not-actually-find-fields/191461/1 "2019-07-20T00:07:09Z")

</div>

(Elasticsearch and filebeat are both v7.2)  
I was encountering a lot of difficulty using the _convert_ processor to change types, so I simplified things down to using _rename_. The results of my tests are confusing.

Given the following config block, I'd expect 6 fields to get renamed:

```
- module: osquery
  result:
    enabled: true
    input:
      processors:
      - rename:
          fields:
          - from: "input.type"
            to: "input.typerenamed"
          - from: "fileset.name"
            to: "fileset.renamed"
          - from: "host.architecture"
            to: "host.architecturerenamed"
          - from: "host.os.platform"
            to: "host.os.platformrenamed"
          - from: "osquery.result.action"
            to: "osquery.result.actionrenamed"
          - from: "osquery.result.columns.actual_rpm"
            to: "osquery.result.columns.actual_rpmrenamed"

```

What I actually get is just 2 fields renamed. _input.type_ and _fileset.name_ both successfully change.

None of the other lines work, instead I get errors saying that the fields were not found. The fields are definitely there, and I even copy/pasted the names just in case I was like unable to spell 'columns' or something 😉

I've tried a number of different fields, some rename, others end up with an error.message in this format:

> Failed to rename fields in processor: could not fetch value for key: _foo_, Error: key not found

I see that there is a "dot expander processor" but it implies that any field with a dot in the name can't be manipulated by a processor, but _input.type_ and _fileset.name_ do...

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [July 22, 2019, 10:50am UTC](https://discuss.elastic.co/t/using-a-processor-in-a-filebeat-module-may-or-may-not-actually-find-fields/191461/2 "2019-07-22T10:50:42Z")

</div>

You set the processor on the module (input) level, when only `input.type` and `fileset.name` fields are present.

`host.*` fields are set on Filebeat level and those are added right before the event gets published. Thus, you cannot rename those fields in the input level, only after `add_host_metadata` processor is applied.

As parsing of osquery input is only done in ES, you cannot rename `osquery.result.*` in Filebeat, as they are not yet present. For reference, this is the event that's published to ES:

The following event is published to ES:

```auto
{
  "@timestamp": "2019-07-22T10:38:38.006Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.0.0",
    "pipeline": "filebeat-8.0.0-osquery-result-pipeline"
  },
  "event": {
    "module": "osquery",
    "dataset": "osquery.result"
  },
  "fileset": {
    "name": "result"
  },
  "service": {
    "type": "osquery"
  },
  "ecs": {
    "version": "1.0.1"
  },
  "host": {
    "os": {
      "name": "Debian GNU/Linux",
      "kernel": "4.9.0-9-amd64",
      "codename": "stretch",
      "platform": "debian",
      "version": "9 (stretch)",
      "family": "debian"
    },
    "id": "505afdafda3b4f33a63749ae39284742",
    "containerized": false,
    "hostname": "sleipnir",
    "name": "sleipnir",
    "architecture": "x86_64"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/home/n/go/src/github.com/elastic/beats/filebeat/module/osquery/result/test/test.log"
    }
  },
  "json": {
    "action": "removed",
    "name": "pack_it-compliance_mounts",
    "unixTime": "1514472008",
    "hostIdentifier": "192-168-0-4.rdsnet.ro",
    "columns": {
      "path": "/private/var/vm",
      "flags": "345018372",
      "inodes_free": "9223372036854775804",
      "blocks": "122061322",
      "blocks_available": "75966945",
      "blocks_free": "121274885",
      "device_alias": "/dev/disk1s4",
      "inodes": "9223372036854775807",
      "blocks_size": "4096",
      "device": "/dev/disk1s4",
      "type": "apfs"
    },
    "calendarTime": "Thu Dec 28 14:40:08 2017 UTC",
    "epoch": "0",
    "counter": "1",
    "decorations": {
      "username": "tsg",
      "host_uuid": "4AB2906D-5516-5794-AF54-86D1D7F533F3"
    }
  },
  "input": {
    "type": "log"
  },
  "agent": {
    "version": "8.0.0",
    "type": "filebeat",
    "ephemeral_id": "8d173703-148f-4abb-bbc8-2dd02c3c80cb",
    "hostname": "sleipnir",
    "id": "e2887f41-5f2c-4f5e-b655-e7142da3386c"
  }
}

```

So if you would like to rename osquery fields, I suggest you edit the Ingest pipeline of osquery module: [https://github.com/elastic/beats/blob/master/filebeat/module/osquery/result/ingest/pipeline.json](https://github.com/elastic/beats/blob/master/filebeat/module/osquery/result/ingest/pipeline.json)

---

<div class="post-metadata">

**Author:** ![dedemorton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dedemorton/32/84409_2.png) [@dedemorton](https://discuss.elastic.co/u/dedemorton)\
**Post date:** [July 22, 2019, 8:49pm UTC](https://discuss.elastic.co/t/using-a-processor-in-a-filebeat-module-may-or-may-not-actually-find-fields/191461/3 "2019-07-22T20:49:03Z")

</div>

FYI that I've created an issue in GitHub to improve the documentation because it's not clear that some fields in the event are not available to Beats processors: [https://github.com/elastic/beats/issues/13023](https://github.com/elastic/beats/issues/13023)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 19, 2019, 10:49pm UTC](https://discuss.elastic.co/t/using-a-processor-in-a-filebeat-module-may-or-may-not-actually-find-fields/191461/4 "2019-08-19T22:49:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.