Logs from logstash indexer produce abnormal spam

Elastic Stack Logstash
1

Dear community,

Logstash version 1.5.6

Problem:
I have some logstash configs populated cross the fleet as well as on logatash indexer itself.
When I simulate error from any agent, event is processed, stored in ES and emailed properly.
When I simulate the same event on indexer it starts sending countless amount of messages with mutated subject with multiply %{process} field.

Config below:

input{
file{
type => "syslog"
path => "/var/log/messages"
start_position => "end"
sincedb_path => "/etc/logstash-sincedb/syslog"
}
}

filter {
if [type] == "syslog" and [host] == "elk.improve" {
grok{
patterns_dir => [ "/etc/logstash/patterns" ]
match => ["message", "%{SYSLOGTIMESTAMP:timestamp} %{HOST} %{WORD:source}: %{WORD:process} ?(.oom-killer.): %{GREEDYDATA:info}"]
add_tag => ["syslog_oom_error"]
}

if "_grokparsefailure" in [tags] {
drop {}
}
}
}

output {
if [type] == "syslog" and [host] == "elk.improve"{
redis {
host => "redis.improve"
data_type => "list"
key => "logstash"
}
}
if "syslog_oom_error" in [tags] and [host] == "elk.improve"{
email{
to => "reciever"
from => "sender"
subject => "Logstash: %{process} invoked OOM-killer on %{host}"
body => "%{message}"
via => "smtp"
options => {
smtpIporHost => "mail.perrit.nl"
port => "25"
userName => "somename"
password => ***
authenticationType => plain
}
}
}

}

Outputs sent with subject

Logstash: sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd invoked OOM-killer on elk.improve

The same messages from rest servers as I mentioned are processing fine. What is wrong with this particular config?

Thanks for any support,
Sergey

2

I stopped logstash on all servers but indexer, and still receive tons of messages. It looks like a bug in sincedb stuff or somehwere...
Does anyone have any clue where to dig?

3

Is your Logstash indexer receiving all of the events from the other Logstash hosts as well?

I'd suggest you add a another output like file or stdout that matches the same messages as your email output so you can log the messages that are causing the spam and see why.

Keep in mind that the Linux OOM output is multiline, not sure if you are interested in all of the lines or not.

4

Now all logstash shippers are stopped, so only indexer is online.
Environment: redis -> logstash indexer -> elasticsearch All are located on single server

I managed to localize the problem but can't explain it:
When I use redis as output on indexer host in any config it ends up with producing tons of messages to email, file or elasticsearch. http://screencast.com/t/mAL34wPSXN
Redis output used in any config:

   redis {
                    host => "redis.improve"
                    data_type => "list"
                    key => "logstash"
            }

Indexer.conf:

input {
redis {
host => "redis.improve"
data_type => "list"
key => "logstash"
codec => json
}
}

filter {

}

output {
elasticsearch {
hosts => "elk.improve"
}
}

Redis output from all other shippers works normal but behaves insane on indexer. How this can be explained?

Thanks ahead,
Sergey

5

So folks,

any ideas why 'redis' output on indexer is botched? Should I use 'localhost' if redis is running next to logstash?

Thanks
Sergey

6

Can anyone at least flag how you manage local logs of indexer while running redis on it?

Thanks
Sergey