Robin_Guo
(Robin Guo)
January 28, 2019, 10:59am
1
Note: the raw PHP larval log is consist of JSON object format text and non-JSON format text
Original PHP larval log:
[2019-01-28 02:28:22] prod.INFO: {"database":"test10","type":"form","id":"20821","comment":"passback:792","timestamp":"2019-01-28 02:28:22"}
log is collected by filebeat ,output to logstash and logstash output to local file
{
"info": "{\"database\":\"test10\",\"type\":\"form\",\"id\":\"19860\",\"comment\":\"passback:792\",\"timestamp\":\"2019-01-28 02:27:39\"}"
}
The output I wanted
{"info":{"database":"test10","type":"form","id":"19860","comment":"passback:792","timestamp":"2019-01-28 02:27:39"}}
logstash conf:
#logstash for pipeline filebeat
input {
beats {
port => 5058
}
}
filter {
if [service] =~ "datapurge-log" {
grok {
match => ["message", "%{UNUSED_TIME:unused_time} %{UNUSED_TYPE:unused_type}\: %{INFO:info}"]
}
mutate {
remove_field=> ["[beat][version]","[beat][hostname]","[host][name]","[input][type]","[prospector][type]","[offset]","[message]"],
gsub => ["info", "[\\]", ""]
}
}
}
output {
if "_grokparsefailure" not in [tags] {
file {
path => "/tmp/datapurge-log"
}
}
Badger
January 28, 2019, 2:32pm
2
I would suggest chopping up the message using a dissect filter, then using a json filter to parse it.
dissect { mapping => { "message" => "[%{} %{}] %{}: %{msg}" } }
Robin_Guo
(Robin Guo)
January 30, 2019, 3:00am
3
Hi @Badger ,
we ship logs like this: filebeat->logstash->ES
when the Original PHP larval log is collected by filebeat, the event is added to the key message . and all hash within the JSON object was added the escape character \ .
my requirement is that remove strings both [2019-01-28 02:28:22] prod.INFO: and escape character \ in the hash of the whole JSON object.
PS:
Before collecting:
[2019-01-28 02:28:22] prod.INFO: {"database":"test10","type":"form","id":"20821","comment":"passback:792","timestamp":"2019-01-28 02:28:22"}
After collecting:
{
"message": "[2019-01-28 02:28:22] prod.INFO: "{\"database\":\"test10\",\"type\":\"form\",\"id\":\"19860\",\"comment\":\"passback:792\",\"timestamp\":\"2019-01-28 02:27:39\"}"}
}
Any help or Suggestion would be appreciate!
Thanks in advance
Badger
January 30, 2019, 1:37pm
4
As I said, use the dissect filter that I showed to parse the message, then use a json filter to parse the field that contains the JSON.
system
(system)
Closed
February 27, 2019, 1:37pm
5
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.