How does Logstash parsing the below events into correct json data?

Elastic Stack Logstash
1

Note: the raw PHP larval log is consist of JSON object format text and non-JSON format text

Original PHP larval log:

[2019-01-28 02:28:22] prod.INFO: {"database":"test10","type":"form","id":"20821","comment":"passback:792","timestamp":"2019-01-28 02:28:22"}

log is collected by filebeat ,output to logstash and logstash output to local file

{
  "info": "{\"database\":\"test10\",\"type\":\"form\",\"id\":\"19860\",\"comment\":\"passback:792\",\"timestamp\":\"2019-01-28 02:27:39\"}"
}

The output I wanted

{"info":{"database":"test10","type":"form","id":"19860","comment":"passback:792","timestamp":"2019-01-28 02:27:39"}}

logstash conf:

#logstash for pipeline filebeat
input {
  beats {
   port => 5058
    }
}

filter {

  if [service] =~ "datapurge-log" {
     grok {
       match => ["message", "%{UNUSED_TIME:unused_time} %{UNUSED_TYPE:unused_type}\: %{INFO:info}"]
     }
     mutate {
       remove_field=> ["[beat][version]","[beat][hostname]","[host][name]","[input][type]","[prospector][type]","[offset]","[message]"],
       gsub => ["info", "[\\]", ""]
     }

   }
}

output {
   if "_grokparsefailure"  not in [tags] {
     file {
       path => "/tmp/datapurge-log"
     }
   }
2

I would suggest chopping up the message using a dissect filter, then using a json filter to parse it.

dissect { mapping => { "message" => "[%{} %{}] %{}: %{msg}" } }
3

Hi @Badger,

we ship logs like this: filebeat->logstash->ES

when the Original PHP larval log is collected by filebeat, the event is added to the key message. and all hash within the JSON object was added the escape character \.
my requirement is that remove strings both [2019-01-28 02:28:22] prod.INFO: and escape character \ in the hash of the whole JSON object.

PS:

Before collecting:

[2019-01-28 02:28:22] prod.INFO: {"database":"test10","type":"form","id":"20821","comment":"passback:792","timestamp":"2019-01-28 02:28:22"}

After collecting：

{
 "message": "[2019-01-28 02:28:22] prod.INFO: "{\"database\":\"test10\",\"type\":\"form\",\"id\":\"19860\",\"comment\":\"passback:792\",\"timestamp\":\"2019-01-28 02:27:39\"}"}
}

Any help or Suggestion would be appreciate！

Thanks in advance

4

As I said, use the dissect filter that I showed to parse the message, then use a json filter to parse the field that contains the JSON.