# Parsing a syslog message before sending to elastic with empty fields

**URL:** <https://discuss.elastic.co/t/parsing-a-syslog-message-before-sending-to-elastic-with-empty-fields/357795>\
**Category:** Logstash\
**Created:** [April 19, 2024, 2:09pm UTC](https://discuss.elastic.co/t/parsing-a-syslog-message-before-sending-to-elastic-with-empty-fields/357795 "2024-04-19T14:09:59Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![devdem13](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@devdem13](https://discuss.elastic.co/u/devdem13)\
**Post date:** [April 19, 2024, 2:09pm UTC](https://discuss.elastic.co/t/parsing-a-syslog-message-before-sending-to-elastic-with-empty-fields/357795/1 "2024-04-19T14:09:59Z")

</div>

Hi,

I am sending syslog data to logstash in the form of attribute=value separated by spaces. Sometime some fields might be empty. For example in the syslog message you might have:

JOBNAME=ABCD SOURCE= CLASS=CLASS1 ENTITY=ENTITY2

I am just using a kv parser to forward to elastic. The SOURCE value and others  
sometimes have no data in them. Thus after forwarding to elastic it gets mixed up on the KV pairs.

Looking on elastic I see:

"JOBNAME" : "ABCD",  
"SOURCE" : "CLASS=CLASS1",  
"ENTITY":"ENTITY2",

So I don't want to lose the CLASS KV pair as it is attached to the SOURCE attribute by mistake.

I have been looking on here and all I have found is examples when the entry is empty after the colon already, not in how to get the entry placed right in elastic.

I think I would like it to show up as:

"JOBNAME" : "ABCD",  
"SOURCE" :""  
"CLASS":"CLASS1",  
"ENTITY":"ENTITY2",

or maybe even

"JOBNAME" : "ABCD",  
"CLASS":"CLASS1",  
"ENTITY":"ENTITY2",

How can I do this generically for any attribute or would I have to put special  
filtering in for each attribute that might be empty?

Thanks,  
Pat

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 19, 2024, 2:58pm UTC](https://discuss.elastic.co/t/parsing-a-syslog-message-before-sending-to-elastic-with-empty-fields/357795/2 "2024-04-19T14:58:18Z")

</div>

Hello,

Please share the logstash configuration you are using.

What is generating those messages? Do you have any control over it to change the format and use double quotes in the values? Unquoted values can mix up the KV filter and you may need to use other filters to parse the message.

---

<div class="post-metadata">

**Author:** ![devdem13](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@devdem13](https://discuss.elastic.co/u/devdem13)\
**Post date:** [April 19, 2024, 4:11pm UTC](https://discuss.elastic.co/t/parsing-a-syslog-message-before-sending-to-elastic-with-empty-fields/357795/3 "2024-04-19T16:11:08Z")

</div>

I have a bunch of messages that are generated by underlying code.  
So I can't easily change the code so I was trying to get logstash to make it work  
when a field was empty.

So it seems everything comes in as gets stored in a field called message.  
My filter is simple right now.

```auto

filter {
   kv { }

   if [type] == "syslog" {
     mutate {
            gsub => ["message", "= ", "=EMPTY "]
    }
  }

```

I was trying to replace "= " with "=EMPTY " but the value was changed in the output of the message field. I need to filter on input maybe if that is possible?  
I am not sure.

My output looks like this right now:

```auto

output {
  file {
    path => "C:\logstash\logstash-8.12.0\logs\log%{type}.%{{yyyy.MM.dd.HH}}"
  }
  elasticsearch {
      hosts => ["localhost:9200"]
      index => "cem_index5" 
      ilm_enabled => false 	
  }
}

```

So it looks like maybe I can put the filter on the input.  
I will try that next.

Pat

---

<div class="post-metadata">

**Author:** ![devdem13](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@devdem13](https://discuss.elastic.co/u/devdem13)\
**Post date:** [April 19, 2024, 7:38pm UTC](https://discuss.elastic.co/t/parsing-a-syslog-message-before-sending-to-elastic-with-empty-fields/357795/4 "2024-04-19T19:38:58Z")

</div>

I can't see anything in the DOC that allows me to replace all occurrences of a string with another. Seems like that grok matches work on a field, but the input string is causing my field to not be found.

Is there a way to replace the "= " on input with a default value?  
I can't seem to figure this out.

Maybe a totally different approach is needed?

Thanks,  
Pat

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 19, 2024, 7:48pm UTC](https://discuss.elastic.co/t/parsing-a-syslog-message-before-sending-to-elastic-with-empty-fields/357795/5 "2024-04-19T19:48:16Z")

</div>

> [@devdem13](#):
>
> I can't see anything in the DOC that allows me to replace all occurrences of a string with another.

That is exactly what mutate+gsub does.

What does your input data look like, and what kv filter configuration are you using?

---

<div class="post-metadata">

**Author:** ![devdem13](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@devdem13](https://discuss.elastic.co/u/devdem13)\
**Post date:** [April 19, 2024, 7:52pm UTC](https://discuss.elastic.co/t/parsing-a-syslog-message-before-sending-to-elastic-with-empty-fields/357795/6 "2024-04-19T19:52:10Z")

</div>

> [@devdem13](#):
>
> ```auto
> filter {
> kv { }
> 
> if [type] == "syslog" {
> mutate {
> gsub => ["message", "= ", "=EMPTY "]
> }
> }
> 
> ```

IT is like above. That mutate gsub only modifies fields, but my fields don't get created properly when there is nothing for a value.

```auto
filter {
   kv { }

   if [type] == "syslog" {
     mutate {
            gsub => ["message", "= ", "=EMPTY "]
    }
  }

```

This would change the message field itself, but doesn't fix where a value assigned to an empty field came from the next kv pair in the data.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 19, 2024, 7:57pm UTC](https://discuss.elastic.co/t/parsing-a-syslog-message-before-sending-to-elastic-with-empty-fields/357795/7 "2024-04-19T19:57:09Z")

</div>

You need to use the mutate before your kv filter to modify the source `message` field.

Just but the `kv` after your conditional.

---

<div class="post-metadata">

**Author:** ![devdem13](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@devdem13](https://discuss.elastic.co/u/devdem13)\
**Post date:** [April 23, 2024, 2:45pm UTC](https://discuss.elastic.co/t/parsing-a-syslog-message-before-sending-to-elastic-with-empty-fields/357795/8 "2024-04-23T14:45:54Z")

</div>

Thanks. That did get it to work.