# Syslog message is incomplete

**URL:** <https://discuss.elastic.co/t/syslog-message-is-incomplete/192384>\
**Category:** Logstash\
**Created:** [July 26, 2019, 8:25am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384 "2019-07-26T08:25:09Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![mondei1](https://avatars.discourse-cdn.com/v4/letter/m/7ba0ec/32.png) [@mondei1](https://discuss.elastic.co/u/mondei1)\
**Post date:** [July 26, 2019, 8:25am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/1 "2019-07-26T08:25:10Z")

</div>

Hey community,  
I have to setup a ELK Stack and everything runs perfectly until I had to implement syslogs. My problem here is that the syslog I receive in Logstash differs from the syslog that I receive with rsyslog (located in /var/log/syslog).

Here is what I found in `/var/log/syslog`:

```auto
Jul 26 09:06:39 hostname.local openvpn[46263]: write UDPv4: No buffer space available (code=55)

```

And here is the message in Logstash:

```auto
<27>Jul 26 09:06:39 openvpn[46263]: write UDPv4: No buffer space available (code=55)

```

No big difference, right? Well, the hostname is missing and I have that weird number at the beginning (tell me If you know what it is). So the grok pattern (see below) is not 100% working.

My filter:

```auto
input {
  tcp {
    port => 5000
    type => syslog
  }
  udp {
    port => 5000
    type => syslog
    codec => plain
  }
# stdin { type => syslog }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_tag => ["syslog"]
    }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
      add_tag => ["apache"]
    }
    date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
      target => "syslog_timestamp"
    }
    date {
      match => ["apache_timestamp", "dd/MMM/YYYY:HH:mm:ss Z","dd/MMM/YYYY:HH:mm:ss Z"]
      target => "apache_timestamp"
    }
    if "beats_input_codec_plain_applied" in [tags] {
      mutate {
        remove_tag => ["beats_input_codec_plain_applied"]
      }
    }
    if [host] == "logserver" {
      mutate {
        replace => ["host", "%{syslog_hostname}"]
        remove_field => ["syslog_hostname"]
      }
    }
  }
}

output {
  elasticsearch {
    hosts => localhost
    user => user
    password => super_secret-CENSORED-pw
    ssl => false
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

```

And here is what I get in return:

```auto
{
          "tags" => [
        [0] "_grokparsefailure"
    ],
       "message" => "<27>Jul 26 09:06:39 openvpn[46263]: write UDPv4: No buffer space available (code=55)",
          "host" => "xx.xxx.x.xx", # which is not the hostname but a ip address that never changes
          "type" => "syslog",
      "@version" => "1",
    "@timestamp" => 2019-07-26T09:06:39.496Z
}

```

Why is there a difference, why isn't it the default syntax of a normal syslog.

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [July 26, 2019, 8:45am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/2 "2019-07-26T08:45:04Z")

</div>

> I have that weird number at the beginning (tell me If you know what it is).

It is syslog priority, see: [RFC 5424 - The Syslog Protocol](https://tools.ietf.org/html/rfc5424#section-6.2.1)

I don't fully understand your setup, would you elaborate it a bit? What is sending the syslog, rsyslog? Logstash is always receiving, not ever sending, right? At least I don't see syslog output.

```auto
    if "beats_input_codec_plain_applied" in [tags] {
      mutate {
        remove_tag => ["beats_input_codec_plain_applied"]
      }
    }

```

BTW, you can replace this by configuring beats input not to add the tag: [Beats input plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-include_codec_tag)

---

<div class="post-metadata">

**Author:** ![mondei1](https://avatars.discourse-cdn.com/v4/letter/m/7ba0ec/32.png) [@mondei1](https://discuss.elastic.co/u/mondei1)\
**Post date:** [July 26, 2019, 8:59am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/3 "2019-07-26T08:59:15Z")

</div>

> [@admlko](#):
>
> It is syslog priority, see: [RFC 5424 - The Syslog Protocol](https://tools.ietf.org/html/rfc5424#section-6.2.1)

> [@admlko](#):
>
> BTW, you can replace this by configuring beats input not to add the tag: [Beats input plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-include_codec_tag)

Thanks, good to know.

> [@admlko](#):
>
> Logstash is always receiving, not ever sending, right?

Right. I have a different servers where different services are running on and server are sending logs to this one logserver with syslog (over udp).

> [@admlko](#):
>
> I don't fully understand your setup, would you elaborate it a bit?

I have two filters, one for generic syslogs and the other one for apache syslogs. And I noticed that If I paste a log in where the hostname is given, the field `host` is the hostname of the logserver, I don't want that, so I replace the value of `host` and delete the `syslog_hostname`.

Thanks for the fast awnser.

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [July 26, 2019, 9:03am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/4 "2019-07-26T09:03:40Z")

</div>

Sorry, I meant that I am not sure about your environment configuration, I understood the Logstash config 🙂  
But now I also understand your environment. So about tackling the hostname issue, the sending client is not sending the hostname as part of the syslog message, right? Then it is a config issue with the client, that's why I was asking about rsyslog.  
Or does Logstash mystically remove that part?

---

<div class="post-metadata">

**Author:** ![mondei1](https://avatars.discourse-cdn.com/v4/letter/m/7ba0ec/32.png) [@mondei1](https://discuss.elastic.co/u/mondei1)\
**Post date:** [July 26, 2019, 9:17am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/5 "2019-07-26T09:17:01Z")

</div>

I try now to use Filebeats which reads out the content of `/var/log/syslog`.

> [@admlko](#):
>
> Then it is a config issue with the client, that's why I was asking about rsyslog.

Well, I ran `tcpdump` and I see here that `syslog` don't even sends the hostname. Syslog adds the hostname afterwards.

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [July 26, 2019, 9:21am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/6 "2019-07-26T09:21:51Z")

</div>

Please be more clear. Syslog is a standard, not a product. Rsyslog and syslog-ng are the most common syslog servers. /var/log/syslog is just a file where one of those servers writes.

So you have /var/log/syslog file, which includes the hostname in the syslog message. But when you read it with Filebeat, which sends it to Logstash, the hostname is missing, correct?

And you checked that Filebeat strips the hostname from message with tcpdump?

---

<div class="post-metadata">

**Author:** ![mondei1](https://avatars.discourse-cdn.com/v4/letter/m/7ba0ec/32.png) [@mondei1](https://discuss.elastic.co/u/mondei1)\
**Post date:** [July 26, 2019, 9:38am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/7 "2019-07-26T09:38:38Z")

</div>

Sorry If I didn't was clear.

Corrently I read the UDP stream from syslog directly with Logstash, process it and sends it to Elasticsearch. But then I noticed that the hostname is missing. Some minutes ago I ran `tcpdump` to inspect the raw packets from syslog. There I noticed that not Logstash removed the hostname but syslog just didn't sent the hostname. Logstash tooks the raw package by UDP.

The logs written in `/var/log/syslog` are modified afterwards where the hostname is written in.

I am now trying to read the syslog with Filebeat where the hostname should be included in Logstash.

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [July 26, 2019, 9:43am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/8 "2019-07-26T09:43:06Z")

</div>

Maybe I am starting to grasp what is going on here. So you have a syslog server, which **both** writes to the /var/log/syslog and then you have configured it to send the events also to Logstash over UDP?  
If this is the case, your syslog server is using different template for writing to the log and sending the event over the network. You have to check the manual of the server.

If you want to just read the /var/log/syslog file with Filebeat, there shouldn't be any issue and you should be able to parse it with Logstash properly, if you just add the syslog priority in your grok pattern (or add a new pattern containing it if you are also receiving syslog messages which doesn't include the priority).

---

<div class="post-metadata">

**Author:** ![mondei1](https://avatars.discourse-cdn.com/v4/letter/m/7ba0ec/32.png) [@mondei1](https://discuss.elastic.co/u/mondei1)\
**Post date:** [July 26, 2019, 10:16am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/9 "2019-07-26T10:16:42Z")

</div>

Well I managed it to get the messages with Filebeat. Previously I used Logstash only to receive the syslog messages.

I can't run the syslog service and Logstash concurrently because of the port. Now, I don't listen to syslog with Logstash anymore, instead I read the syslog file with Filebeat which gives me the hostname. Filebeat sends it now to Logstash where I can filter it.

So I think the issue is resolved now.

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [July 26, 2019, 10:20am UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/10 "2019-07-26T10:20:17Z")

</div>

I still don't understand the second paragraph at all (where are all those components and services running on), but if you are happy with it, then great 🙂

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 26, 2019, 12:32pm UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/11 "2019-07-26T12:32:38Z")

</div>

> [@mondei1](#):
>
> I have that weird number at the beginning (tell me If you know what it is)

Note that logstash has a syslog\_pri filter that will parse that number into its component fields.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 23, 2019, 12:32pm UTC](https://discuss.elastic.co/t/syslog-message-is-incomplete/192384/12 "2019-08-23T12:32:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.