Parsing a syslog message before sending to elastic with empty fields

Elastic Stack Logstash
1

Hi,

I am sending syslog data to logstash in the form of attribute=value separated by spaces. Sometime some fields might be empty. For example in the syslog message you might have:

JOBNAME=ABCD SOURCE= CLASS=CLASS1 ENTITY=ENTITY2

I am just using a kv parser to forward to elastic. The SOURCE value and others
sometimes have no data in them. Thus after forwarding to elastic it gets mixed up on the KV pairs.

Looking on elastic I see:

"JOBNAME" : "ABCD",
"SOURCE" : "CLASS=CLASS1",
"ENTITY":"ENTITY2",

So I don't want to lose the CLASS KV pair as it is attached to the SOURCE attribute by mistake.

I have been looking on here and all I have found is examples when the entry is empty after the colon already, not in how to get the entry placed right in elastic.

I think I would like it to show up as:

"JOBNAME" : "ABCD",
"SOURCE" :""
"CLASS":"CLASS1",
"ENTITY":"ENTITY2",

or maybe even

"JOBNAME" : "ABCD",
"CLASS":"CLASS1",
"ENTITY":"ENTITY2",

How can I do this generically for any attribute or would I have to put special
filtering in for each attribute that might be empty?

Thanks,
Pat

2

Hello,

Please share the logstash configuration you are using.

What is generating those messages? Do you have any control over it to change the format and use double quotes in the values? Unquoted values can mix up the KV filter and you may need to use other filters to parse the message.

3

I have a bunch of messages that are generated by underlying code.
So I can't easily change the code so I was trying to get logstash to make it work
when a field was empty.

So it seems everything comes in as gets stored in a field called message.
My filter is simple right now.


filter {
   kv { }

   if [type] == "syslog" {
     mutate {
            gsub => [ "message", "= ", "=EMPTY " ]
    }
  }

I was trying to replace "= " with "=EMPTY " but the value was changed in the output of the message field. I need to filter on input maybe if that is possible?
I am not sure.

My output looks like this right now:


output {
  file {
    path => "C:\logstash\logstash-8.12.0\logs\log%{type}.%{{yyyy.MM.dd.HH}}"
  }
  elasticsearch {
      hosts => ["localhost:9200"]
      index => "cem_index5" 
      ilm_enabled => false 	
  }
}

So it looks like maybe I can put the filter on the input.
I will try that next.

Pat

4

I can't see anything in the DOC that allows me to replace all occurrences of a string with another. Seems like that grok matches work on a field, but the input string is causing my field to not be found.

Is there a way to replace the "= " on input with a default value?
I can't seem to figure this out.

Maybe a totally different approach is needed?

Thanks,
Pat

5

That is exactly what mutate+gsub does.

What does your input data look like, and what kv filter configuration are you using?

6

IT is like above. That mutate gsub only modifies fields, but my fields don't get created properly when there is nothing for a value.

filter {
   kv { }

   if [type] == "syslog" {
     mutate {
            gsub => [ "message", "= ", "=EMPTY " ]
    }
  }

This would change the message field itself, but doesn't fix where a value assigned to an empty field came from the next kv pair in the data.

7

You need to use the mutate before your kv filter to modify the source message field.

Just but the kv after your conditional.

8

Thanks. That did get it to work.