Multilne codec merging UDP log entries


#1

I am using the multiline codec in Logstash 5.0. For the most part it works, merging entries that consist of several lines. So far all true multiline entries are merged correctly. The issue is that some random entries are being merged as well.

     udp {
            host => "172.17.220.148"
            port => 514
            type => "netapp"
            codec => multiline {
                   pattern => "^\s"
                   what => "previous"
            }

Here is an example of merging that should not occur:

Source
<191>Jan 12 08:03:55 [host:api_mpool_06:debug]: sumnerTunnel:command is exec /sbin/asup_opt -priv 'admin' -get 'autosupport.enable':command exit status: 0
<191>Jan 12 08:03:55 [host:api_mpool_06:debug]: sumnerTunnel:command is exec /sbin/asup_opt -priv 'admin' -get 'autosupport.enable':mhost_exec_internal return status: 0

after grok:
sumnerTunnel:command is exec /sbin/asup_opt -priv 'admin' -get 'autosupport.enable':command exit status: 0 <191>Jan 12 08:03:55 [host:api_mpool_06:debug]: sumnerTunnel:command is exec /sbin/asup_opt -priv 'admin' -get 'autosupport.enable':mhost_exec_internal return status: 0

grok works fine. I do see some grok errors but the original message is blank. I looked at both the previews and next entry and they look good.

To add some scope a 6 day sampling consists of:
405257 entries total
36 entries with grok errors (the entries do not contain any information so safe to drop)
245 entries that contain 2 entries in one
All true multi-line entries work (that I looked at)

I remembering reading something about a buffer causing this but could not find it again.


(Christian Dahlqvist) #2

UDP does not guarantee delivery or ordering so I would if possible recommend doing multiline processing before sending the data via UDP.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.