# Elasticsearch throws exceptions while using with logstash and kibana **URL:** \ **Category:** Elasticsearch\ **Created:** [April 1, 2013, 12:32pm UTC](https://discuss.elastic.co/t/elasticsearch-throws-exceptions-while-using-with-logstash-and-kibana/11397 "2013-04-01T12:32:09Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Sunny\_Jaisinghani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunny_jaisinghani/32/2398_2.png) [@Sunny\_Jaisinghani](https://discuss.elastic.co/u/Sunny_Jaisinghani)\ **Post date:** [April 1, 2013, 12:32pm UTC](https://discuss.elastic.co/t/elasticsearch-throws-exceptions-while-using-with-logstash-and-kibana/11397/1 "2013-04-01T12:32:09Z")
Hello Experts, I am new to elasticsearch and i am using elasticsearch in embedded mode with logstash. I have recently started exploring logstash after i heard it from my peers who attended puppetconf 2012 We have deployed logstash 1.1.9 in standalone mode with embedded elasticsearch on a EC2 instance. Elasticsearch version is 0.20.2 We have also installed kibana as a frontend for logstash. Since, we are using the embedded version of elasticsearch, i assume it doesn't have any compatibility issues with logstash version. {:message=\>"Failed to index an event, will retry", :exception=\>org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [1m], :event=\>{"@source"=\>"file://ip-10-xx-xx-xx.ec2.internal/var/log/messages", "@tags"=\>[], "@fields"=\>{}, "@timestamp"=\>"2013-03-01T09:08:39.998Z", "@source\_host"=\>"ip-10-157-xx-xx.ec2.internal", "@source\_path"=\>"/var/log/messages", "@message"=\>"Mar 1 00:14:52 ip-10-122-70-221@type"=\>"linuxsyslog"}, :level=\>:warn} I am also seeing following exception {:message=\>"Failed to index an event, will retry", :exception=\>org.elasticsearch.transport.RemoteTransportException: [Imperial Hydra][inet[/10.157.38.34:9300]][index], :event=\>{"@source"=\>"file://ip-10-xx-xx-xx.ec2.internal/var/log/", "@tags"=\>[], "@fields"=\>{}, "@timestamp"=\>"2013-02-27T13:49:13.515Z", "@source\_host"=\>"ip-10-xx-xx-xx.ec2.internal", "@source\_path"=\>"/var/log/", {:message=\>"Failed to index an event, will retry", :exception=\>org.elasticsearch.transport.RemoteTransportException: [Ariann] {:message=\>"Failed to index an event, will retry", :exception=\>org.elasticsearch.transport.RemoteTransportException: [Gorgon] {:message=\>"Failed to index an event, will retry", :exception=\>org.elasticsearch.transport.RemoteTransportException: [Forgotten One] following processes are running on the host root 22155 1 1 00:11 pts/0 00:01:52 /usr/bin/java -jar /opt/logstash/bin/logstash.jar agent --config /opt/logstash/conf/logstash.conf --log /opt/logstash/log/logstash.log --grok-patterns-path /opt/logstash/patterns root 22039 1 0 00:08 ? 00:00:01 ruby kibana.rb root 28932 13009 0 02:05 pts/0 00:00:00 grep -i java tcp 0 0 0.0.0.0:9200 0.0.0.0:\* LISTEN 22155/java tcp 0 0 0.0.0.0:80 0.0.0.0:\* LISTEN 22039/ruby tcp 0 0 0.0.0.0:9300 0.0.0.0:\* LISTEN 22155/java tcp 0 0 0.0.0.0:9301 0.0.0.0:\* LISTEN 22155/java # logstash.conf input { file { type =\> "linuxsyslog" ``` # Wildcards work, here :) path => ["/var/log/messages"] sincedb_path => "/opt/logstash" } file { type => "merrors" path => ["/var/log/m.errors*"] sincedb_path => "/opt/logstash" } file { type => "mall" path => ["/var/log/m*"] exclude => ["/var/log/m.errors*"] sincedb_path => "/opt/logstash" } ``` } filter { ``` date { type => "syslog" syslog_timestamp => ["MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } ``` } output { elasticsearch { embedded =\> true host =\> "10.157.3x.3x" } } If i restart logstash and try to use the search engine, it works and gives output for a while, before it starts complaining about the above errors. # curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' { "cluster\_name" : "elasticsearch", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 2, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 26, "active\_shards" : 26, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 34 } Also, i am not sure why the above output shows "number of nodes" = 2 I am using logstash and elasticsearch in standalone mode. The above output was taken when everything was running smoothly. I am running these on an small EC2 instance running RHEL 5.4 . ( 1.7 GB RAM ) java version is 1.6 Please suggest on how to fix these errors. Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Sunny\_Jaisinghani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunny_jaisinghani/32/2398_2.png) [@Sunny\_Jaisinghani](https://discuss.elastic.co/u/Sunny_Jaisinghani)\ **Post date:** [April 1, 2013, 12:38pm UTC](https://discuss.elastic.co/t/elasticsearch-throws-exceptions-while-using-with-logstash-and-kibana/11397/2 "2013-04-01T12:38:22Z")
Here are the exceptions i am seeing :exception=\>org.elasticsearch.action.UnavailableShardsException: :exception=\>org.elasticsearch.discovery.MasterNotDiscoveredException: :exception=\>org.elasticsearch.transport.RemoteTransportException: On Monday, 1 April 2013 18:02:09 UTC+5:30, Sunny Jaisinghani wrote: > Hello Experts, > > I am new to elasticsearch and i am using elasticsearch in embedded mode > with logstash. > > I have recently started exploring logstash after i heard it from my peers > who attended puppetconf 2012 > > We have deployed logstash 1.1.9 in standalone mode with embedded > elasticsearch on a EC2 instance. > Elasticsearch version is 0.20.2 > We have also installed kibana as a frontend for logstash. > Since, we are using the embedded version of elasticsearch, i assume it > doesn't have any compatibility issues with logstash version. > > {:message=\>"Failed to index an event, will retry", > :exception=\>org.elasticsearch.discovery.MasterNotDiscoveredException: > waited for [1m], > :event=\>{"@source"=\>"file://ip-10-xx-xx-xx.ec2.internal/var/log/messages", > "@tags"=\>, "@fields"=\>{}, "@timestamp"=\>"2013-03-01T09:08:39.998Z", > "@source\_host"=\>"ip-10-157-xx-xx.ec2.internal", > "@source\_path"=\>"/var/log/messages", "@message"=\>"Mar 1 00:14:52 > ip-10-122-70-221@type"=\>"linuxsyslog"}, :level=\>:warn} > > I am also seeing following exception > > {:message=\>"Failed to index an event, will retry", > :exception=\>org.elasticsearch.transport.RemoteTransportException: [Imperial > Hydra][inet[/10.157.38.34:9300]][index], > :event=\>{"@source"=\>"file://ip-10-xx-xx-xx.ec2.internal/var/log/", > "@tags"=\>, "@fields"=\>{}, "@timestamp"=\>"2013-02-27T13:49:13.515Z", > "@source\_host"=\>"ip-10-xx-xx-xx.ec2.internal", "@source\_path"=\>"/var/log/", > > {:message=\>"Failed to index an event, will retry", > :exception=\>org.elasticsearch.transport.RemoteTransportException: [Ariann] > {:message=\>"Failed to index an event, will retry", > :exception=\>org.elasticsearch.transport.RemoteTransportException: [Gorgon] > {:message=\>"Failed to index an event, will retry", > :exception=\>org.elasticsearch.transport.RemoteTransportException: > [Forgotten One] > > following processes are running on the host > > root 22155 1 1 00:11 pts/0 00:01:52 /usr/bin/java -jar > /opt/logstash/bin/logstash.jar agent --config > /opt/logstash/conf/logstash.conf --log /opt/logstash/log/logstash.log > --grok-patterns-path /opt/logstash/patterns > root 22039 1 0 00:08 ? 00:00:01 ruby kibana.rb > > root 28932 13009 0 02:05 pts/0 00:00:00 grep -i java > tcp 0 0 0.0.0.0:9200 > 0.0.0.0:\* LISTEN 22155/java > tcp 0 0 0.0.0.0:80 > 0.0.0.0:\* LISTEN 22039/ruby > tcp 0 0 0.0.0.0:9300 > 0.0.0.0:\* LISTEN 22155/java > tcp 0 0 0.0.0.0:9301 > 0.0.0.0:\* LISTEN 22155/java > > # logstash.conf > > input { > file { > type =\> "linuxsyslog" > > ``` > # Wildcards work, here :) > path => ["/var/log/messages"] > sincedb_path => "/opt/logstash" > } > > file { > type => "merrors" > path => ["/var/log/m.errors*"] > sincedb_path => "/opt/logstash" > } > > file { > type => "mall" > path => ["/var/log/m*"] > exclude => ["/var/log/m.errors*"] > sincedb_path => "/opt/logstash" > } > > ``` > > } > > filter { > > ``` > date { > type => "syslog" > syslog_timestamp => ["MMM d HH:mm:ss", "MMM dd HH:mm:ss"] > } > > ``` > > } > > output { > elasticsearch { > embedded =\> true > host =\> "10.157.3x.3x" > } > } > > If i restart logstash and try to use the search engine, it works and gives > output for a while, before it starts complaining about the above errors. > > # curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > > { > "cluster\_name" : "elasticsearch", > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 2, > "number\_of\_data\_nodes" : 1, > "active\_primary\_shards" : 26, > "active\_shards" : 26, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 34 > } > > Also, i am not sure why the above output shows "number of nodes" = 2 > I am using logstash and elasticsearch in standalone mode. > The above output was taken when everything was running smoothly. > > I am running these on an small EC2 instance running RHEL 5.4 . ( 1.7 GB > RAM ) > java version is 1.6 > > Please suggest on how to fix these errors. > > Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Sunny\_Jaisinghani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunny_jaisinghani/32/2398_2.png) [@Sunny\_Jaisinghani](https://discuss.elastic.co/u/Sunny_Jaisinghani)\ **Post date:** [April 2, 2013, 4:56am UTC](https://discuss.elastic.co/t/elasticsearch-throws-exceptions-while-using-with-logstash-and-kibana/11397/3 "2013-04-02T04:56:29Z")
When elasticsearch hangs, the curl request does not give any output # curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' and i keep seeing these exceptions :exception=\>org.elasticsearch.discovery.MasterNotDiscoveredException: However; connections to port 9200 are getting through. # telnet 127.0.0.1 9200 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. ^] telnet\> quit Connection closed. On Monday, 1 April 2013 18:02:09 UTC+5:30, Sunny Jaisinghani wrote: > Hello Experts, > > I am new to elasticsearch and i am using elasticsearch in embedded mode > with logstash. > > I have recently started exploring logstash after i heard it from my peers > who attended puppetconf 2012 > > We have deployed logstash 1.1.9 in standalone mode with embedded > elasticsearch on a EC2 instance. > Elasticsearch version is 0.20.2 > We have also installed kibana as a frontend for logstash. > Since, we are using the embedded version of elasticsearch, i assume it > doesn't have any compatibility issues with logstash version. > > {:message=\>"Failed to index an event, will retry", > :exception=\>org.elasticsearch.discovery.MasterNotDiscoveredException: > waited for [1m], > :event=\>{"@source"=\>"file://ip-10-xx-xx-xx.ec2.internal/var/log/messages", > "@tags"=\>, "@fields"=\>{}, "@timestamp"=\>"2013-03-01T09:08:39.998Z", > "@source\_host"=\>"ip-10-157-xx-xx.ec2.internal", > "@source\_path"=\>"/var/log/messages", "@message"=\>"Mar 1 00:14:52 > ip-10-122-70-221@type"=\>"linuxsyslog"}, :level=\>:warn} > > I am also seeing following exception > > {:message=\>"Failed to index an event, will retry", > :exception=\>org.elasticsearch.transport.RemoteTransportException: [Imperial > Hydra][inet[/10.157.38.34:9300]][index], > :event=\>{"@source"=\>"file://ip-10-xx-xx-xx.ec2.internal/var/log/", > "@tags"=\>, "@fields"=\>{}, "@timestamp"=\>"2013-02-27T13:49:13.515Z", > "@source\_host"=\>"ip-10-xx-xx-xx.ec2.internal", "@source\_path"=\>"/var/log/", > > {:message=\>"Failed to index an event, will retry", > :exception=\>org.elasticsearch.transport.RemoteTransportException: [Ariann] > {:message=\>"Failed to index an event, will retry", > :exception=\>org.elasticsearch.transport.RemoteTransportException: [Gorgon] > {:message=\>"Failed to index an event, will retry", > :exception=\>org.elasticsearch.transport.RemoteTransportException: > [Forgotten One] > > following processes are running on the host > > root 22155 1 1 00:11 pts/0 00:01:52 /usr/bin/java -jar > /opt/logstash/bin/logstash.jar agent --config > /opt/logstash/conf/logstash.conf --log /opt/logstash/log/logstash.log > --grok-patterns-path /opt/logstash/patterns > root 22039 1 0 00:08 ? 00:00:01 ruby kibana.rb > > root 28932 13009 0 02:05 pts/0 00:00:00 grep -i java > tcp 0 0 0.0.0.0:9200 > 0.0.0.0:\* LISTEN 22155/java > tcp 0 0 0.0.0.0:80 > 0.0.0.0:\* LISTEN 22039/ruby > tcp 0 0 0.0.0.0:9300 > 0.0.0.0:\* LISTEN 22155/java > tcp 0 0 0.0.0.0:9301 > 0.0.0.0:\* LISTEN 22155/java > > # logstash.conf > > input { > file { > type =\> "linuxsyslog" > > ``` > # Wildcards work, here :) > path => ["/var/log/messages"] > sincedb_path => "/opt/logstash" > } > > file { > type => "merrors" > path => ["/var/log/m.errors*"] > sincedb_path => "/opt/logstash" > } > > file { > type => "mall" > path => ["/var/log/m*"] > exclude => ["/var/log/m.errors*"] > sincedb_path => "/opt/logstash" > } > > ``` > > } > > filter { > > ``` > date { > type => "syslog" > syslog_timestamp => ["MMM d HH:mm:ss", "MMM dd HH:mm:ss"] > } > > ``` > > } > > output { > elasticsearch { > embedded =\> true > host =\> "10.157.3x.3x" > } > } > > If i restart logstash and try to use the search engine, it works and gives > output for a while, before it starts complaining about the above errors. > > # curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > > { > "cluster\_name" : "elasticsearch", > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 2, > "number\_of\_data\_nodes" : 1, > "active\_primary\_shards" : 26, > "active\_shards" : 26, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 34 > } > > Also, i am not sure why the above output shows "number of nodes" = 2 > I am using logstash and elasticsearch in standalone mode. > The above output was taken when everything was running smoothly. > > I am running these on an small EC2 instance running RHEL 5.4 . ( 1.7 GB > RAM ) > java version is 1.6 > > Please suggest on how to fix these errors. > > Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:43am UTC](https://discuss.elastic.co/t/elasticsearch-throws-exceptions-while-using-with-logstash-and-kibana/11397/4 "2017-07-06T02:43:18Z")