Filebeat 7.1 UDP + JSON

[Serg]

Hi
I try to use such config to get data from my app. on UDP socket in JSON format.
I hope it will help to mark all fields automatically (like in GELF driver)

filebeat.inputs:

- type: udp
  host: "localhost:9099"
  max_message_size: 100KiB

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:
  host: "1.1.1.1:5601"


output.elasticsearch:

  hosts: ["1.1.1.2:9200"]
  indices:
    - index: "system-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "system"
    - index: "drupal-%{[agent.version]}-%{+yyyy.MM.dd}"


processors:
  - add_cloud_metadata: ~
  - add_host_metadata:
      netinfo.enabled: true
      cache.ttl: 5m
  - decode_json_fields:
      fields: ["message"]
      process_array: true
      max_depth: 10
      overwrite_keys: true

But I've got such kind of error in filebeat logs, could plz tell what I am doing wrong?

Private:interface {}(nil)}, Flags:0x1} (status=400):

{"type":"mapper_parsing_exception","reason":"failed to parse field [message] of type [text] in document with id 'JGNk_m0BojXHgnHnNPHC'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:366"}}

From app side for integration I use monolog, it could send valid JSON data

[pmercado (PM)]

Hi @Serg,

can this be that the index mapping already exists as text at elasticsearch, and you are trying to use it as an object (after being JSON decoded)?

Can you retry using target element at your configuration to write to a non already mapped field?
https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html