Is UDP input JSON aware?


(Maurorappa) #1

I'm having trouble using the UDP input in the same way the file input does with Json files.
I would like automatic field recognition for the structured data it receives.
Here's my test:
I'm using version 6.2.1 (amd64), libbeat 6.2.1.
Bare minimum config file:

filebeat.prospectors:

  • type: udp
    enabled: true
    json.add_error_key: true
    json.keys_under_root: true

output.console:
pretty: true

processors:

  • decode_json_fields:
    fields: ['reads', 'writes']
    process_array: false
    max_depth: 1

basic test program in golang:

package main
import (
"net"
"fmt"
)
func main() {
//Connect udp
conn, err := net.Dial("udp", "127.0.0.1:8080")
if err != nil {
//return err
}
fmt.Print("connected")
defer conn.Close()
//simple write
conn.Write([]byte("{"reads":3,"writes":1}\n"))
}

the json data are simply included in the message field and not split in singular values.

2018-02-27T13:47:32.209Z DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-02-27T13:47:32.209Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.1"
},
"beat": {
"hostname": "eurvlii06649",
"version": "6.2.1",
"name": "eurvlii06649"
},
"message": "{"reads":3,"writes":1}\n",
"prospector": {
"type": "udp"
}
}
{
"@timestamp": "2018-02-27T13:47:32.209Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.1"
},
"message": "{"reads":3,"writes":1}\n",
"prospector": {
"type": "udp"
},
"beat": {
"hostname": "eurvlii06649",
"version": "6.2.1",
"name": "eurvlii06649"
}
}

the message is correct as you can see with this test using shell:

[lc20583@eurvlii06649 filebeat-6.2.1-linux-x86_64]$ sudo nc -ul 127.0.0.1 8080 | jq
{
"reads": 3,
"writes": 1
}

I also tried to remove the whole processor block, but got the same result.
The source code for the udp input has far less function/lines of code compared to the log file, is json parsing not available?

Thanks for any reply!


(ruflin) #2

The udp input does not have the json configs. But you can use the processor.

How do you send the data to the udp input? Is all json on one line or spread across multiple lines?


(Maurorappa) #3

it's one line, I put the reproducer code in the opening comment to clarify.
Anyway, I'll write a patch if I have time :wink:


(ruflin) #4

Not 100% sure I can follow. Discussing it on a PR sounds great, code should clarify things :slight_smile:


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.