# Logs from logstash indexer produce abnormal spam

**URL:** <https://discuss.elastic.co/t/logs-from-logstash-indexer-produce-abnormal-spam/37935>\
**Category:** Logstash\
**Created:** [December 24, 2015, 11:20am UTC](https://discuss.elastic.co/t/logs-from-logstash-indexer-produce-abnormal-spam/37935 "2015-12-24T11:20:04Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![szemlyanoy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/szemlyanoy/32/22173_2.png) [@szemlyanoy](https://discuss.elastic.co/u/szemlyanoy)\
**Post date:** [December 24, 2015, 11:20am UTC](https://discuss.elastic.co/t/logs-from-logstash-indexer-produce-abnormal-spam/37935/1 "2015-12-24T11:20:04Z")

</div>

Dear community,

Logstash version 1.5.6

Problem:  
I have some logstash configs populated cross the fleet as well as on logatash indexer itself.  
When I simulate error from any agent, event is processed, stored in ES and emailed properly.  
When I simulate the same event on indexer it starts sending countless amount of messages with mutated subject with multiply %{process} field.

Config below:

input{  
file{  
type =\> "syslog"  
path =\> "/var/log/messages"  
start\_position =\> "end"  
sincedb\_path =\> "/etc/logstash-sincedb/syslog"  
}  
}

filter {  
if [type] == "syslog" and [host] == "elk.improve" {  
grok{  
patterns\_dir =\> ["/etc/logstash/patterns"]  
match =\> ["message", "%{SYSLOGTIMESTAMP:timestamp} %{HOST} %{WORD:source}: %{WORD:process} ?(._oom-killer._): %{GREEDYDATA:info}"]  
add\_tag =\> ["syslog\_oom\_error"]  
}

if "\_grokparsefailure" in [tags] {  
drop {}  
}  
}  
}

output {  
if [type] == "syslog" and [host] == "elk.improve"{  
redis {  
host =\> "redis.improve"  
data\_type =\> "list"  
key =\> "logstash"  
}  
}  
if "syslog\_oom\_error" in [tags] and [host] == "elk.improve"{  
email{  
to =\> "reciever"  
from =\> "sender"  
subject =\> "Logstash: %{process} invoked OOM-killer on %{host}"  
body =\> "%{message}"  
via =\> "smtp"  
options =\> {  
smtpIporHost =\> "[mail.perrit.nl](http://mail.perrit.nl)"  
port =\> "25"  
userName =\> "somename"  
password =\> \*\*\*  
authenticationType =\> plain  
}  
}  
}

}

Outputs sent with subject

Logstash: sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd,sssd invoked OOM-killer on elk.improve

The same messages from rest servers as I mentioned are processing fine. What is wrong with this particular config?

Thanks for any support,  
Sergey

---

<div class="post-metadata">

**Author:** ![szemlyanoy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/szemlyanoy/32/22173_2.png) [@szemlyanoy](https://discuss.elastic.co/u/szemlyanoy)\
**Post date:** [December 30, 2015, 8:50pm UTC](https://discuss.elastic.co/t/logs-from-logstash-indexer-produce-abnormal-spam/37935/2 "2015-12-30T20:50:06Z")

</div>

I stopped logstash on all servers but indexer, and still receive tons of messages. It looks like a bug in sincedb stuff or somehwere...  
Does anyone have any clue where to dig?

---

<div class="post-metadata">

**Author:** ![Joshua\_Rich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshua_rich/32/44953_2.png) [@Joshua\_Rich](https://discuss.elastic.co/u/Joshua_Rich)\
**Post date:** [December 30, 2015, 11:00pm UTC](https://discuss.elastic.co/t/logs-from-logstash-indexer-produce-abnormal-spam/37935/3 "2015-12-30T23:00:59Z")

</div>

Is your Logstash indexer receiving all of the events from the other Logstash hosts as well?

I'd suggest you add a another output like `file` or `stdout` that matches the same messages as your `email` output so you can log the messages that are causing the spam and see why.

Keep in mind that the Linux OOM output is multiline, not sure if you are interested in all of the lines or not.

---

<div class="post-metadata">

**Author:** ![szemlyanoy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/szemlyanoy/32/22173_2.png) [@szemlyanoy](https://discuss.elastic.co/u/szemlyanoy)\
**Post date:** [January 1, 2016, 3:26pm UTC](https://discuss.elastic.co/t/logs-from-logstash-indexer-produce-abnormal-spam/37935/4 "2016-01-01T15:26:17Z")

</div>

Now all logstash shippers are stopped, so only indexer is online.  
Environment: redis -\> logstash indexer -\> elasticsearch All are located on single server

I managed to localize the problem but can't explain it:  
When I use redis as output on indexer host in any config it ends up with producing tons of messages to email, file or elasticsearch. [http://screencast.com/t/mAL34wPSXN](http://screencast.com/t/mAL34wPSXN)  
Redis output used in any config:

```
   redis {
                    host => "redis.improve"
                    data_type => "list"
                    key => "logstash"
            }

```

**Indexer.conf:**

input {  
redis {  
host =\> "redis.improve"  
data\_type =\> "list"  
key =\> "logstash"  
codec =\> json  
}  
}

filter {

}

output {  
elasticsearch {  
hosts =\> "elk.improve"  
}  
}

Redis output from all other shippers works normal but behaves insane on indexer. How this can be explained?

Thanks ahead,  
Sergey

---

<div class="post-metadata">

**Author:** ![szemlyanoy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/szemlyanoy/32/22173_2.png) [@szemlyanoy](https://discuss.elastic.co/u/szemlyanoy)\
**Post date:** [January 6, 2016, 8:58am UTC](https://discuss.elastic.co/t/logs-from-logstash-indexer-produce-abnormal-spam/37935/5 "2016-01-06T08:58:12Z")

</div>

So folks,

any ideas why 'redis' output on indexer is botched? Should I use 'localhost' if redis is running next to logstash?

Thanks  
Sergey

---

<div class="post-metadata">

**Author:** ![szemlyanoy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/szemlyanoy/32/22173_2.png) [@szemlyanoy](https://discuss.elastic.co/u/szemlyanoy)\
**Post date:** [January 18, 2016, 8:07pm UTC](https://discuss.elastic.co/t/logs-from-logstash-indexer-produce-abnormal-spam/37935/6 "2016-01-18T20:07:12Z")

</div>

Can anyone at least flag how you manage local logs of indexer while running redis on it?

Thanks  
Sergey

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:15am UTC](https://discuss.elastic.co/t/logs-from-logstash-indexer-produce-abnormal-spam/37935/7 "2017-07-06T05:15:18Z")

</div>

Closed on July 6, 2017, 5:15am UTC