Filebeat 6.3.1 UDP + Json

Elastic Stack Beats
1

Hi,

I'm trying to send a JSON Log over UDP, so filebeat forwards it to the ElasticSearch.
The thing is, I can't get it to decode the JSON payload

I also tried with a \n and a \n\n to no luck

{
  "@timestamp": "2018-07-18T21:11:57.266Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.1",
    "truncated": false
  },
  "input": {
    "type": "udp"
  },
  "prospector": {
    "type": "udp"
  },
  "beat": {
    "name": "dad20104e72c",
    "hostname": "dad20104e72c",
    "version": "6.3.1",
    "timezone": "+00:00"
  },
  "host": {
    "name": "dad20104e72c",
    "architecture": "x86_64",
    "os": {
      "family": "debian",
      "codename": "xenial",
      "platform": "ubuntu",
      "version": "16.04.4 LTS (Xenial Xerus)"
    },
    "id": "b987bfc2295c4e82afddb796793d7ac1",
    "containerized": true
  },
  "source": "127.0.0.1:40783",
  "message": "{\"inner\":\"John\",\"outer\":30}"
}

Python Script

import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(bytes('{"inner":"John","outer":30}','utf-8'), ('127.0.0.1', 7070))

Netcat Debug

nc -ul 127.0.0.1 -p 7070
{"inner":"John","outer":30}

Filebeat Version

filebeat version 6.3.1 (amd64), libbeat 6.3.1 [ed42bb85e72ae58cc09748dc1825159713e0ffd4 built 2018-06-29 21:09:35 +0000 UTC]

Python 3 Version

Python 3.6.5 (default, May  3 2018, 10:08:28)
[GCC 5.4.0 20160609] on linux

Filebeat .yml

filebeat.prospectors:
- type: udp
  max_message_size: 10KiB
  host: "localhost:7070"
  enabled: true
  processors:
  - add_locale: ~
  - add_host_metadata: ~
  - decode_json_fields:
      fields: ["inner", "outer"]

output.console:
  pretty: true

logging.level: debug
2

I see, the issue here is some confusion over the decode_json_fields option. That option specifies what the source field is that you want to decode. If you look at the output you posted above, the JSON is all in the message field. You'll need to use a config like:

filebeat.prospectors:
- type: udp
  max_message_size: 10KiB
  host: "localhost:7070"
  enabled: true
  processors:
  - add_locale: ~
  - add_host_metadata: ~
  - decode_json_fields:
      fields: ["message"]

output.console:
  pretty: true

logging.level: debug

That will decode the fields. You may also want to rename those fields once they're decoded.

3

Hi @Andrew_Cholakian1

You are completely right ! , it's working as expect maybe I recommend the presence of an example on the Decode Json Fields page.

Thank your for your time :slight_smile:
Antonio

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.