Filebeat Drop field doesnt work for json

Elastic Stack Beats
1 
filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - "*.json"
processors:
- drop_fields:
    fields: ["value"]
``````````````````````````````

That's my filebeat yml file. the drop fields doesn't work, any suggestions?
2

Hi @niv, welcome to the Elastic Community Forums!

Can you post a sample event that's being published by Filebeat with this configuration? To do so it might be convenient to temporarily change your Filebeat output to console.

Also, just guessing based on your your log file paths setting, I wonder if you need to use the json setting under your log input configuration?

Shaunak

3

heres my JSON :

[
    {
        "value":  "dropfields_3",
        "PSN":  "dropfields_3",
        "Id":  "dropfields_3",
        "PSShowComputerName":  "dropfields_3"
    }
]


filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - "*.json"
        scan_frequency: 1s
        tail_files: false
        multiline.pattern: '{'
        multiline.negate: true
        multiline.match: after
        clean_removed: true
        ignore_older: 336h
        close_inactive: 24h

processors:
      - drop_fields:
        fields: ["value"]
4

HERE"S THE OUTPUT ON CONSOLE :

{
  "@timestamp": "2020-09-15T18:15:45.372Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.3.0"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "sample.json"
    }
  },
  "message": "[",
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.0.1"
  },
  "host": {
    "name": "xxxxxxxxxxxxxxxxxxxxxxxxx"
  },
  "agent": {
    "ephemeral_id": "xxxxxxxxxxxxxxxxxxxx",
    "hostname": "xxxxxxxxxxxxx",
    "id": "xxxxxxxxxxxxxxxxxxxxx",
    "version": "7.3.0",
    "type": "filebeat"
  }
}
{
  "@timestamp": "2020-09-15T18:15:45.372Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.3.0"
  },
  "log": {
    "flags": [
      "multiline"
    ],
    "offset": 3,
    "file": {
      "path": "sample.json"
    }
  },
  "message": "    {\n        \"value\":  \"dropfields_3\",\n        \"PSN\":  \"dropfields_3\",\n        \"Id\":  \"dropfields_3\",\n        \"PSShowComputerName\":  \"drop
fields_3\"\n    }",
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "x.x.x"
  },
  "host": {
    "name": "xxxxxxxxxxxxxxxxxx"
  },
  "agent": {
    "version": "7.3.0",
    "type": "filebeat",
    "ephemeral_id": "xxxxxxxxxxxxxxxxxx",
    "hostname": "xxxxxxxxxxxxxxxxxxxx",
    "id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  }
}

I also dont want the extra fields to appear.

5

Hi, please edit your posts and enclose your code/config/log snippets in ``` delimiters so we can read them more easily with the indentation in place.

Thanks,

Shaunak

6

Done, I also need help to remove the unwanted fields that filebeat adds like @version, @metadata, agent, etc. I tried "fields_under_root : false" but that didn't work .

7

Is your entire log file one JSON array containing multiple JSON objects (one object per log entry) or is it one JSON array containing one JSON object (one array per log entry)?

Shaunak

8

that was just a sample file I was using to test. Typically it would look like this.


[
    {
        "value":  "dropfields_3",
        "PSN":  "dropfields_3",
        "Id":  "dropfields_3",
        "PSShowComputerName":  "dropfields_3"
    },
	    {
        "value":  "dropfields_1",
        "PSN":  "dropfields_1",
        "Id":  "dropfields_1",
        "PSShowComputerName":  "dropfields_1"
    },
	    {
        "value":  "dropfields_2",
        "PSN":  "dropfields_2",
        "Id":  "dropfields_2",
        "PSShowComputerName":  "dropfields_2"
    }
]
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.