Generic filter for Messages containing JSON

Elastic Stack Logstash
1

Hi,
I have the following initial situation, we get various log messages which have JSON content, they look like this, for example:

message:Log Text case1 #json: {"case1-total":26,"case-valid":8,"case-duplicates":16,"case-violations":{"Size":0,"Duration":2,"Length":0,"Speed":0}}

message:Log Text case2 #json: {"case2-processed":10,"case-created":10,"case-updated":0}

message:Log Text case3 #json: {"case3-total":10,"case-valid":7,"case-violations":{"missing":["AAA","BBB"],"Data":["AAA","BBB","CCC"]}}

In order to get all fields out of the messages cleanly, I have a filter like this one for each case:

if "Log Text case1 #json:" in [message] {
  mutate { add_field => { "case" => "%%{message}" } }
  mutate { gsub => [ "case", "^.* ", "" ] }
}

This creates the following fields, for example:

case.case1-duplicates:16
case.case1-total:26
case.case1-valid:8
case.case1-violations.Size:0
case.case1-violations.Duration:2
case.case1-violations.Length:0
case.case1-violations.Speed:0

I find that extremely annoying, since the Logstash filters have to be expanded with every system extension, so I thought it had to work more generically, e.g. with such a filter:

if "#json" in [message] {
  mutate { add_field => { "json_message" => "%%{message}" } }
  mutate { gsub => [ "json_message", "^.* ", "" ] }
}

This creates the following fields, for example:

json_message.case1-duplicates:16
json_message.case1-total:26
json_message.case1-valid:8
json_message.case1-violations.Size:0
json_message.case1-violations.Duration:2
json_message.case1-violations.Length:0
json_message.case1-violations.Speed:0

What bothers me, however, is that I have all the cases in the "json_message" field. It would be better if I didn't have to create the "json_message" field at all and could only generate the individual fields from the JSON strings.
For example, the following would be acceptable as a result:

case1-duplicates:16
case1-total:26
case1-valid:8
case1-violations.Size:0
case1-violations.Duration:2
case1-violations.Length:0
case1-violations.Speed:0

I'm probably just too stupid and don't see the solution.
Does anyone here have an idea?

Regards
Thorsten

2

Hi,

I'm not sure what you really want to do.
What does "%%{message}" mean?

If you meant "%{message}", more simple way is as belows (only if the message contains single json and contains no extra {}s), though I just check it by grok debugger and not in Logstash.

filter {
  grok{
    match => {"message" => "(?<json>\{.*\})"}
  }
  json {
    source => "json"
    remove_field => ["json", "message"]
  }
}
1 Like
3

Hi,

sorry that was a typo with "%%{message}", "%{message}" is the correct value.

The problem is, that also messages with extra "{}s" like

message:Log Text case1 #json: {"case1-total":26,"case-valid":8,"case-duplicates":16,"case-violations":{"Size":0,"Duration":2,"Length":0,"Speed":0}}

could be catched.

4

intended to mean extra {}s outside the json.
I suppose the above filter can cope with your example.

5

Sorry for the late respones, I was to bussy.
Your suggestion is working fine :slight_smile:
Thanks a lot

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.