Parsing stringified JSON from filebeat input

(Srinivas K) #1


I am a total newbie to Elastic and co, so please educate. I have filebeat set up to export JSON from a log file with lines like this: "08-Apr-2019 01:11:16.047 INFO [http-nio-443-exec-37] MyClass.Func MyClassFunc() :: InfoObject {action: "start", ID: "a1b2c3", state: "STARTED", time: 1554685876047}".

I am trying to get either filebeat to export the embedded stringified JSON, or get Logstash to decode it into such JSON for further use. So far no luck on either one :frowning:

What I have so far:

  1. Filebeat config:


  • type: log
    enabled: true
    • /home/cheeni/Dev/wordly/datainfra/usage-analysis/fbinput.log


  • dissect:
    tokenizer: "%{Date} %{TS} %{Loglevel} %{Processid} %{Class} %{Function} %{Junk} %{ObjectHead} %{MyObject}"
    field: "message"
    target_prefix: ""

  • include_fields:
    fields: ["host", "source", "MyObject"]

hosts: ["localhost:5044"]
pretty: true

Filebeat output looks like this:

2019-04-15T13:13:30.844-0700 DEBUG [publish] pipeline/processor.go:308 Publish event: {
[... @timestamp and @metadata stuff..]
"MyObject": "{action: "start", ID: "a1b2c3", state: "STARTED", time: 1554685876047}",
"host": {
"name": "Mymachine"

I'd like to get the stringified JSON value of "MyObject" to be proper JSON:
"MyObject": {
action: "start",
ID: "a1b2c3",
state: "STARTED",
time: 1554685876047

  1. Logstash config:

input {
beats {
codec => "json"
port => "5044"

filter {
json {
source => "message"
target => "newObj"

output {
stdout { codec => rubydebug }

The logstash json filter seems to have no effect on MyObject, and it just spits out the same field and string value that filebeat sends it.

Any guidance?



You shouldn't need a json filter if you have a json codec. What does an event look like in rubydebug?

(Srinivas K) #3

Pretty much the same as the filebeat output, as this shows:

"MyObject" => "{action: "start", ID: "a1b2c3", state: "STARTED", time: 1554686173129}",
"@version" => "1",
"@timestamp" => 2019-04-15T20:13:30.844Z,
"tags" => [
[0] "beats_input_raw_event"



json {
     source => "MyObject"
     target => "newObj"

(Srinivas K) #5

I did try that in the logstash conf, but it resulted in a _jsonparsefailure. I am clearly doing something wrong. Is there a way to make logstash output verbose so that there is some detail for the json parse failure?

"MyObject" => "{action: "update", ID: "a1b2c3", state: "END", time: 1554687389174}",
"@timestamp" => 2019-04-16T00:02:28.173Z,
"@version" => "1",
"tags" => [
[0] "beats_input_raw_event",
[1] "_jsonparsefailure"


That is not valid JSON. Where you have action you should have "action". If necessary you can make that happen with a mutate filter.

Please blockquote your data and configurations. Edit your post and select the text you need to quote and click on </> in the toolbar above the edit pane.

(Srinivas K) #7

Ouch! Indeed that is not valid JSON. Sorry I wasted time!