# Logstash not indexing anything **URL:** \ **Category:** Logstash\ **Created:** [July 3, 2015, 7:33am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844 "2015-07-03T07:33:59Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 7:33am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/1 "2015-07-03T07:33:59Z")
Hello! I thought I had Logstash all figured out but apparently I don't. It worked fine for me up until today. It indexed the log files I wanted to but today it just won't index anything at all and in the log I keep getting the error message "message=\>"retrying failed action with response code: 503", :level=\>:warn}" I have searched around a bit and I have no idea what is going on. ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 7:57am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/2 "2015-07-03T07:57:39Z")
Apparently it can read what happened from yesterday and the days before that but not today. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2015, 8:32am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/3 "2015-07-03T08:32:16Z")
How's the health of you ES cluster? Anything interesting in the logs? ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 9:14am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/5 "2015-07-03T09:14:04Z")
The logs are saying stuff like "Too many attempts at sending events. Dropping "event name" This is what it looks like stash\_1 | {:timestamp=\>"2015-07-03T09:12:12.540000+0000", :message=\>"too many attempts at sending event. dropping: 2015-07-03T09:05:53.514Z 1100e328950e 2015-0 7-03 09:05:53,039+0000 DEBUG [qtp2092176926-68027] \*UNKNOWN org.sonatype.nexus.content.internal.ContentAuthenticationFilter - Attempting to authenticate Subject as Anonymous request...", :level=\>:error} ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 9:28am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/6 "2015-07-03T09:28:19Z")
How do you see the health of your cluster? After some research my best guess is that the logstash has stashed to many events or something. Is there a way to fix this? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2015, 10:49am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/7 "2015-07-03T10:49:22Z")
The [cluster health API](https://www.elastic.co/guide/en/elasticsearch/guide/current/cluster-health.html) is useful, and you should definitely run an ES dashboard plugin like kopf. Sorry for being unclear about the logs; it's the ES logs I'm talking about. ES is the software that's misbehaving so it's likely that its logs will be the most interesting. ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 10:55am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/8 "2015-07-03T10:55:32Z")
Where do I find the ES logs naturally? My best guess is that the index is full? Because the log message says that it attempts to send events but fails. There are a lot of words I dont' really know about like "index, indices" and such. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2015, 11:07am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/9 "2015-07-03T11:07:02Z")
> [@simonrisberg](#): > > Where do I find the ES logs naturally? My best guess is that the index is full? Because the log message says that it attempts to send events but fails. The ES logs are normally found in /var/log/elasticsearch. There's no such things as a full index but you may very well have run out of RAM. > There are a lot of words I dont' really know about like "index, indices" and such. I suggest you read [Elasticsearch: The Definitive Guide](https://www.elastic.co/guide/en/elasticsearch/guide/master/index.html) (at least the intro chapter(s)). The [Elasticsearch Reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) document also contains explanations of indexes, shards, and so on. Without a basic understanding of those concepts this is going to be a bumpy ride for you. ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 11:09am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/10 "2015-07-03T11:09:05Z")
Thank you. ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 11:45am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/11 "2015-07-03T11:45:26Z")
The log is saying {:timestamp=\>"2015-07-03T11:42:00.292000+0000", :message=\>"retrying failed action with response code: 503", :level=\>:warn} and {:timestamp=\>"2015-07-03T11:45:04.858000+0000", :message=\>"too many attempts at sending event. dropping: 2015-07-03T11:35:58.600Z 7738b91a24ce Using data from old metadata for com/jeppesen/jcms/ioserver.sdk/19.2.0/ioserver.sdk-19.2.0.rpm", :level=\>:error} ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2015, 11:46am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/12 "2015-07-03T11:46:44Z")
That's the Logstash logfile. What's in the Elasticsearch logfile? ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 11:47am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/13 "2015-07-03T11:47:24Z")
Because of the error message I get it feels like it has stashed away the events from all the logs that I've indexed. Is there some way to remove the events that I've stashed? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2015, 11:50am UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/14 "2015-07-03T11:50:18Z")
Nothing is being "stashed". If Logstash can't send data to its output it halts the pipeline. ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 12:29pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/15 "2015-07-03T12:29:38Z")
Okey! From what the log says. What do you think might be the problem with the pipeline? Have you ever encountered something like this before? I've read up a little bit on shards, nodes and clusters. ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 12:46pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/16 "2015-07-03T12:46:20Z")
I managed to get the status on my cluster and yes it's red. "cluster\_name" : "elk", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 51, "active\_shards" : 51, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 61, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0 ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2015, 1:01pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/17 "2015-07-03T13:01:51Z")
Again, the Elasticsearch logs should contain clues as to what's up with the cluster. There's no point in guessing what it could be. ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 1:04pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/18 "2015-07-03T13:04:31Z")
Remind me, where could I find the elasticsearch logfile? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2015, 1:08pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/19 "2015-07-03T13:08:18Z")
See earlier in the thread. ---
**Author:** ![simonrisberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonrisberg/32/3513_2.png) [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Post date:** [July 3, 2015, 1:08pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/20 "2015-07-03T13:08:56Z")
Apparently the elasticsearch folder is empty. What is the log called so I can search for it? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2015, 1:10pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844/21 "2015-07-03T13:10:55Z")
/var/log/elasticsearch is empty? Is your disk full? [Next page](https://discuss.elastic.co/t/logstash-not-indexing-anything/24844.md?page=2)