Using a processor in a filebeat module may or may not actually find fields

[DPattee]

(Elasticsearch and filebeat are both v7.2)
I was encountering a lot of difficulty using the convert processor to change types, so I simplified things down to using rename. The results of my tests are confusing.

Given the following config block, I'd expect 6 fields to get renamed:

- module: osquery
  result:
    enabled: true
    input:
      processors:
      - rename:
          fields:
          - from: "input.type"
            to: "input.typerenamed"
          - from: "fileset.name"
            to: "fileset.renamed"
          - from: "host.architecture"
            to: "host.architecturerenamed"
          - from: "host.os.platform"
            to: "host.os.platformrenamed"
          - from: "osquery.result.action"
            to: "osquery.result.actionrenamed"
          - from: "osquery.result.columns.actual_rpm"
            to: "osquery.result.columns.actual_rpmrenamed"

What I actually get is just 2 fields renamed. input.type and fileset.name both successfully change.

None of the other lines work, instead I get errors saying that the fields were not found. The fields are definitely there, and I even copy/pasted the names just in case I was like unable to spell 'columns' or something

I've tried a number of different fields, some rename, others end up with an error.message in this format:

Failed to rename fields in processor: could not fetch value for key: foo, Error: key not found

I see that there is a "dot expander processor" but it implies that any field with a dot in the name can't be manipulated by a processor, but input.type and fileset.name do...

[kvch (Noémi Ványi)]

You set the processor on the module (input) level, when only input.type and fileset.name fields are present.

host.* fields are set on Filebeat level and those are added right before the event gets published. Thus, you cannot rename those fields in the input level, only after add_host_metadata processor is applied.

As parsing of osquery input is only done in ES, you cannot rename osquery.result.* in Filebeat, as they are not yet present. For reference, this is the event that's published to ES:

The following event is published to ES:

{
  "@timestamp": "2019-07-22T10:38:38.006Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.0.0",
    "pipeline": "filebeat-8.0.0-osquery-result-pipeline"
  },
  "event": {
    "module": "osquery",
    "dataset": "osquery.result"
  },
  "fileset": {
    "name": "result"
  },
  "service": {
    "type": "osquery"
  },
  "ecs": {
    "version": "1.0.1"
  },
  "host": {
    "os": {
      "name": "Debian GNU/Linux",
      "kernel": "4.9.0-9-amd64",
      "codename": "stretch",
      "platform": "debian",
      "version": "9 (stretch)",
      "family": "debian"
    },
    "id": "505afdafda3b4f33a63749ae39284742",
    "containerized": false,
    "hostname": "sleipnir",
    "name": "sleipnir",
    "architecture": "x86_64"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/home/n/go/src/github.com/elastic/beats/filebeat/module/osquery/result/test/test.log"
    }
  },
  "json": {
    "action": "removed",
    "name": "pack_it-compliance_mounts",
    "unixTime": "1514472008",
    "hostIdentifier": "192-168-0-4.rdsnet.ro",
    "columns": {
      "path": "/private/var/vm",
      "flags": "345018372",
      "inodes_free": "9223372036854775804",
      "blocks": "122061322",
      "blocks_available": "75966945",
      "blocks_free": "121274885",
      "device_alias": "/dev/disk1s4",
      "inodes": "9223372036854775807",
      "blocks_size": "4096",
      "device": "/dev/disk1s4",
      "type": "apfs"
    },
    "calendarTime": "Thu Dec 28 14:40:08 2017 UTC",
    "epoch": "0",
    "counter": "1",
    "decorations": {
      "username": "tsg",
      "host_uuid": "4AB2906D-5516-5794-AF54-86D1D7F533F3"
    }
  },
  "input": {
    "type": "log"
  },
  "agent": {
    "version": "8.0.0",
    "type": "filebeat",
    "ephemeral_id": "8d173703-148f-4abb-bbc8-2dd02c3c80cb",
    "hostname": "sleipnir",
    "id": "e2887f41-5f2c-4f5e-b655-e7142da3386c"
  }
}

So if you would like to rename osquery fields, I suggest you edit the Ingest pipeline of osquery module: https://github.com/elastic/beats/blob/master/filebeat/module/osquery/result/ingest/pipeline.json

[dedemorton]

FYI that I've created an issue in GitHub to improve the documentation because it's not clear that some fields in the event are not available to Beats processors: https://github.com/elastic/beats/issues/13023