Drop_fields not working on doc

Elastic Stack Beats
1

i'm trying to use drop_fields on part of a document:

  "signatures": [
    {
      "markcount": 2,
      "families": [],
      "description": "The binary likely contains encrypted or compressed data indicative of a packer",
      "severity": 2,
      "marks": [
        {
          "entropy": 7.90446336568116,
          "section": {
            "size_of_data": "0x00083400",
            "virtual_address": "0x0036d000",
            "entropy": 7.90446336568116,
            "name": "UPX1",
            "virtual_size": "0x00084000"
          },
          "type": "generic",
          "description": "A section with a high entropy has been found"
        },
        {
          "entropy": 0.9980988593155894,
          "type": "generic",
          "description": "Overall entropy of this PE file is high"
        }
      ],'

specifically, on signatures.marks.section, however my drop_fields configuration does not work:

 - drop_fields:
     fields: ["signatures.marks"]

i have tried signatures.marks.section, signatures.marks, signatures.marks etc to no avail. any ideas?

2

hey @brain

can you check the structure of whole reported event?
is the message you mentioned already json decoded or is it as a string in a source.message field of the event.

3

the message is already json decoded. dropping the signatures field entirely works, but i'd prefer not to do that

4

the problem is that you have an array of signatures
you have few options here.

first and ugly one is to provide enough remove entries so it will look like this

-  drop_event
    fields: ["signatures.[0].marks","signatures.[1].marks","signatures.[2].marks", etc.]

or use log stash (if you have one) and use ruby filter as discussed here: Logstash filter mutate remove_field inside of an array

5

no logstash in the pipeline, so it'll be the ugly solution. how would that work with signatures.marks.section, which is also in an array?

signatures.[0].marks.[0].section, signatures.[0].marks.[1].section, etc..?

6

unfortunately yes. it depends on how many signatures you have in an array, and how many marks are inside a signature
you need to go through [0,0],[0,1]...[0, marks], [1,0]... and so on

unfortunately there's no nice way of doing it in a beat. for more advanced processing using logstash is advised

8

actually, spoke too soon. as a test i've configured:

- drop_fields:
    fields: ["signatures.[0].marks","signatures.[1].marks","signatures.[2].marks","signatures.[3].marks","signatures.[4].marks","signatures.[5].marks","signatures.[6].marks"]
    ignore_missing: true

however i'm still seeing the marks object in the doc, and the doc in question only has one signature field:

cat report.json | jq .signatures

[
  {
    "markcount": 1,
    "families": [],
    "description": "The executable uses a known packer",
    "severity": 1,
    "marks": [
      {
        "category": "packer",
        "ioc": "Armadillo v1.xx - v2.xx",
        "type": "ioc",
        "description": null
      }
    ],
    "references": [],
    "name": "peid_packer"
  }
]
9

hey @brain
I played with this a bit and I found out that beats does not support index pointers (something.0.somethinelse) this is supported only in LogStash
you can specify path up to the point where you hit an array. having array as the first thing in your object is a bit unfortunate.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.