Hi all,
I have a problem when using logstash to get tweets and inserting them into ES. When I stop logstash, via Ctrl+C or Kill (logstash stop doesn’t do anything) makes the indexes where the tweets are being stored to go into red status.
A quick summary of my environment: (all in one VM)
Logstash 1.5.0.rc2
Elasticsearch 1.4.4
Kibana 4.0.1 build 5930
When I run logstash with “/opt/logstash/bin/logstash -f twitter.conf” everything goes fine: tweets are collected and I can see them via Kibana. Taking a lot to the health of indexes:
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-2015.06.08 1 0 43 0 565.9kb 565.9kb
yellow open .marvel-2015.05.23 1 1 8663 0 9.8mb 9.8mb
yellow open .marvel-2015.06.05 1 1 5137 0 6.3mb 6.3mb
green open logstash-2015.06.05 1 0 75 0 600.2kb 600.2kb
yellow open .marvel-2015.05.25 1 1 15146 0 17.9mb 17.9mb
yellow open logstash-2015.03.19 1 1 9082 0 7.7mb 7.7mb
yellow open .marvel-kibana 1 1 1 2 3kb 3kb
green open logstash-2015.06.06 1 0 205835 0 547.7mb 547.7mb
yellow open .marvel-2015.05.22 1 1 400 0 751.9kb 751.9kb
yellow open .marvel-2015.05.26 1 1 30668 0 36mb 36mb
yellow open .marvel-2015.06.06 1 1 29811 0 32.2mb 32.2mb
yellow open .marvel-2015.06.08 1 1 8628 0 19mb 19mb
green open logstash-2015.02.05 1 0 10035 0 10.4mb 10.4mb
yellow open .kibana 1 1 21 0 50.5kb 50.5kb
I highlighted the indexes that contains tweets. Older indexes contain network logs (other data sources… I am building this VM with different ELK use cases and creating different dashboards for each type of data).
The “twitter.conf” file is very simple:
input {
twitter {
consumer_key => "xxxxx"
consumer_secret => "xxxxxx"
oauth_token => "xxxxx"
oauth_token_secret => "xxxxx"
keywords => ["#ChampionsLeagueFinal","BarcelonavsJuventus"]
type => "twitter"
full_tweet => true
}
} #end input block
output {
elasticsearch {
protocol => "node"
node_name => "logstash"
cluster => "test-elastic"
host => "127.0.0.1"
embedded => true
}
} #end output block
When I see I have received enough number of tweets I try to stop logstash, in this case with Ctrl+c and then I see this warning from Logstash:
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed
And when I kill logstash process, automatically, all the indexes that contain tweets (even older indexes) go into a red state:
health status index pri rep docs.count docs.deleted store.size pri.store.size
red open logstash-2015.06.08 1 0
yellow open .marvel-2015.05.23 1 1 8663 0 9.8mb 9.8mb
yellow open .marvel-2015.06.05 1 1 5137 0 6.3mb 6.3mb
red open logstash-2015.06.05 1 0
yellow open .marvel-2015.05.25 1 1 15146 0 17.9mb 17.9mb
yellow open logstash-2015.03.19 1 1 9082 0 7.7mb 7.7mb
yellow open .marvel-kibana 1 1 1 2 3kb 3kb
red open logstash-2015.06.06 1 0
yellow open .marvel-2015.05.22 1 1 400 0 751.9kb 751.9kb
yellow open .marvel-2015.05.26 1 1 30668 0 36mb 36mb
yellow open .marvel-2015.06.06 1 1 29811 0 32.2mb 32.2mb
yellow open .marvel-2015.06.08 1 1 9332 0 19.9mb 19.9mb
green open logstash-2015.02.05 1 0 10035 0 10.4mb 10.4mb
yellow open .kibana 1 1 21 0 50.5kb 50.5kb
Any idea why? Did I do something wrong?
Thanks and best regards,
Rodrigo.