Parsing stringified JSON from filebeat input

Elastic Stack Logstash
1

Greetings,

I am a total newbie to Elastic and co, so please educate. I have filebeat set up to export JSON from a log file with lines like this: "08-Apr-2019 01:11:16.047 INFO [http-nio-443-exec-37] MyClass.Func MyClassFunc() :: InfoObject {action: "start", ID: "a1b2c3", state: "STARTED", time: 1554685876047}".

I am trying to get either filebeat to export the embedded stringified JSON, or get Logstash to decode it into such JSON for further use. So far no luck on either one :frowning:

What I have so far:

  1. Filebeat config:

filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • /home/cheeni/Dev/wordly/datainfra/usage-analysis/fbinput.log

processors:

  • dissect:
    tokenizer: "%{Date} %{TS} %{Loglevel} %{Processid} %{Class} %{Function} %{Junk} %{ObjectHead} %{MyObject}"
    field: "message"
    target_prefix: ""

  • include_fields:
    fields: ["host", "source", "MyObject"]

output.logstash:
hosts: ["localhost:5044"]
codec.json:
pretty: true

Filebeat output looks like this:

2019-04-15T13:13:30.844-0700 DEBUG [publish] pipeline/processor.go:308 Publish event: {
[... @timestamp and @metadata stuff..]
"MyObject": "{action: "start", ID: "a1b2c3", state: "STARTED", time: 1554685876047}",
"host": {
"name": "Mymachine"
}
}

I'd like to get the stringified JSON value of "MyObject" to be proper JSON:
"MyObject": {
action: "start",
ID: "a1b2c3",
state: "STARTED",
time: 1554685876047
}

  1. Logstash config:

input {
beats {
codec => "json"
port => "5044"
}
}

filter {
json {
source => "message"
target => "newObj"
}
}

output {
stdout { codec => rubydebug }
}

The logstash json filter seems to have no effect on MyObject, and it just spits out the same field and string value that filebeat sends it.

Any guidance?

Thanks

2

You shouldn't need a json filter if you have a json codec. What does an event look like in rubydebug?

3

Pretty much the same as the filebeat output, as this shows:

{
"MyObject" => "{action: "start", ID: "a1b2c3", state: "STARTED", time: 1554686173129}",
"@version" => "1",
"@timestamp" => 2019-04-15T20:13:30.844Z,
"tags" => [
[0] "beats_input_raw_event"
]
}

4

Try

json {
     source => "MyObject"
     target => "newObj"
}
5

I did try that in the logstash conf, but it resulted in a _jsonparsefailure. I am clearly doing something wrong. Is there a way to make logstash output verbose so that there is some detail for the json parse failure?

{
"MyObject" => "{action: "update", ID: "a1b2c3", state: "END", time: 1554687389174}",
"@timestamp" => 2019-04-16T00:02:28.173Z,
"@version" => "1",
"tags" => [
[0] "beats_input_raw_event",
[1] "_jsonparsefailure"
]
}

6

That is not valid JSON. Where you have action you should have "action". If necessary you can make that happen with a mutate filter.

Please blockquote your data and configurations. Edit your post and select the text you need to quote and click on </> in the toolbar above the edit pane.

7

Ouch! Indeed that is not valid JSON. Sorry I wasted time!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.