Indexes in ES in red status when I kill logstash

Elastic Stack Elasticsearch
1

Hi all,

I have a problem when using ElasticSearch to index tweets received from Logstash. When I stop logstash, via Ctrl+C or Kill (logstash stop doesn’t do anything) makes the indexes where the tweets are being stored to go into red status.

A quick summary of my environment: (all in one VM)
Logstash 1.5.0.rc2
Elasticsearch 1.4.4
Kibana 4.0.1 build 5930

When I run logstash with “/opt/logstash/bin/logstash -f twitter.conf” everything goes fine: tweets are collected and I can see them via Kibana. Taking a lot to the health of indexes:

health status index pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-2015.06.08 1 0 43 0 565.9kb 565.9kb
yellow open .marvel-2015.05.23 1 1 8663 0 9.8mb 9.8mb
yellow open .marvel-2015.06.05 1 1 5137 0 6.3mb 6.3mb
green open logstash-2015.06.05 1 0 75 0 600.2kb 600.2kb
yellow open .marvel-2015.05.25 1 1 15146 0 17.9mb 17.9mb
yellow open logstash-2015.03.19 1 1 9082 0 7.7mb 7.7mb
yellow open .marvel-kibana 1 1 1 2 3kb 3kb
green open logstash-2015.06.06 1 0 205835 0 547.7mb 547.7mb
yellow open .marvel-2015.05.22 1 1 400 0 751.9kb 751.9kb
yellow open .marvel-2015.05.26 1 1 30668 0 36mb 36mb
yellow open .marvel-2015.06.06 1 1 29811 0 32.2mb 32.2mb
yellow open .marvel-2015.06.08 1 1 8628 0 19mb 19mb
green open logstash-2015.02.05 1 0 10035 0 10.4mb 10.4mb
yellow open .kibana 1 1 21 0 50.5kb 50.5kb

I highlighted the indexes that contains tweets. Older indexes contain network logs (other data sources… I am building this VM with different ELK use cases and creating different dashboards for each type of data).

The “twitter.conf” file is very simple:
input {
twitter {
consumer_key => "xxxxx"
consumer_secret => "xxxxxx"
oauth_token => "xxxxx"
oauth_token_secret => "xxxxx"
keywords => ["#ChampionsLeagueFinal","BarcelonavsJuventus"]
type => "twitter"
full_tweet => true
}
} #end input block

output {
elasticsearch {
protocol => "node"
node_name => "logstash"
cluster => "test-elastic"
host => "127.0.0.1"
embedded => true
}
} #end output block

When I see I have received enough number of tweets I try to stop logstash, in this case with Ctrl+c and then I see this warning from Logstash:

^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed

And when I kill logstash process, automatically, all the indexes that contain tweets (even older indexes) go into a red state:

health status index pri rep docs.count docs.deleted store.size pri.store.size
red open logstash-2015.06.08 1 0
yellow open .marvel-2015.05.23 1 1 8663 0 9.8mb 9.8mb
yellow open .marvel-2015.06.05 1 1 5137 0 6.3mb 6.3mb
red open logstash-2015.06.05 1 0
yellow open .marvel-2015.05.25 1 1 15146 0 17.9mb 17.9mb
yellow open logstash-2015.03.19 1 1 9082 0 7.7mb 7.7mb
yellow open .marvel-kibana 1 1 1 2 3kb 3kb
red open logstash-2015.06.06 1 0
yellow open .marvel-2015.05.22 1 1 400 0 751.9kb 751.9kb
yellow open .marvel-2015.05.26 1 1 30668 0 36mb 36mb
yellow open .marvel-2015.06.06 1 1 29811 0 32.2mb 32.2mb
yellow open .marvel-2015.06.08 1 1 9332 0 19.9mb 19.9mb
green open logstash-2015.02.05 1 0 10035 0 10.4mb 10.4mb
yellow open .kibana 1 1 21 0 50.5kb 50.5kb

Any idea why? Did I do something wrong?

Thanks and best regards,
Rodrigo.

2

Hi Rodrigo,

What is your stack setup?

The embedded flag in logstash is set to true meaning that when you kill logstash, you are killing the embedded elasticsearch instance also.

Looking at you index output, i suspect elasticsearch is complaining as you have a (seperate to logstash) elasticsearch node running so that node knows about the index (replica 0) but does not have the data. Hence the unhealthy index status when you shut down logstash.

Maybe use the nodes API to debug this (and verify what nodes are in the cluster when logstash is running/not). Then you can use the shards api to figure out where the nodes live. If this is the problem, maybe increase the replica count to 1 from 0 :smile:

3

Thank you very much Derry!

You helped me a lot. I eliminated the line "embedded => true", but this alone is not necessary. The real help was to increate the replica count to 1. I don't know how/why I had all other indexes with replica count to 1 but those tweets-indexes with replica 0, but I changed it and it works.

Thanks a lot!

Best regards,
Rodrigo.

4

No problem :wink:

Defaults are normally 5 shards / 1 replica. I'm not sure if Marvel has it's own custom setup.

In general, i would use either embedded with just a cluster size of 1 or else an 'external to logstash' cluster.