Logtsash output to ElasticSearch Cluster

Do i need to set anything up special to have logstash output to my
elasticsearch cluster?

output {
stdout { }
elasticsearch {
cluster => "es-cluster"
}
}

I have my logstash output setup this way but I do not get anything in
kibana - when I switch to elasticsearch_http it works fine and data shows up

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1a4094e7-2457-4790-9a0e-5cec5bb7b12c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Is the cluster external to the LS instance?
Check the logs of your ES master server, it should show the logstash node
join, if you aren't seeing that then enable verbose mode in LS and you
should see the issue.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 27 September 2014 04:28, Kevin M mcgkev29@gmail.com wrote:

Do i need to set anything up special to have logstash output to my
elasticsearch cluster?

output {
stdout { }
elasticsearch {
cluster => "es-cluster"
}
}

I have my logstash output setup this way but I do not get anything in
kibana - when I switch to elasticsearch_http it works fine and data shows up

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1a4094e7-2457-4790-9a0e-5cec5bb7b12c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1a4094e7-2457-4790-9a0e-5cec5bb7b12c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Z1TSCWq%2BvbfMP%2BZCnOSPzPvdX7yY7RxqXbiz5PNZ%2Bb4g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

so it is setup as

Server1
Logstash ES Kibana
server 2
ES only

When I go to the elasticHQ website I see the 2 nodes joined up and healthy

I will check Logstash thanks

On Saturday, September 27, 2014 4:34:45 AM UTC-4, Mark Walkom wrote:

Is the cluster external to the LS instance?
Check the logs of your ES master server, it should show the logstash node
join, if you aren't seeing that then enable verbose mode in LS and you
should see the issue.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 27 September 2014 04:28, Kevin M <mcgk...@gmail.com <javascript:>>
wrote:

Do i need to set anything up special to have logstash output to my
elasticsearch cluster?

output {
stdout { }
elasticsearch {
cluster => "es-cluster"
}
}

I have my logstash output setup this way but I do not get anything in
kibana - when I switch to elasticsearch_http it works fine and data shows up

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1a4094e7-2457-4790-9a0e-5cec5bb7b12c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1a4094e7-2457-4790-9a0e-5cec5bb7b12c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/55c1549d-b115-4257-ab67-8219c3c965c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

So here is the really, i think, relevant log message I got:

log4j, [2014-09-29T08:35:22.061] WARN: org.elasticsearch.discovery:
[Asmodeus] waited for 30s and no initial state was set by the discovery

then a bit later:

log4j, [2014-09-29T08:35:22.061] WARN: org.elasticsearch.discovery:
[Asmodeus] waited for 30s and no initial state was set by the discovery

log4j, [2014-09-29T08:36:27.385] WARN:
org.elasticsearch.discovery.zen.ping.multicast: [Asmodeus] failed to
read requesting data from
/172.16.40.29:54328
java.io.IOException: Expected handle header, got [13]
at
org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:65)
at
org.elasticsearch.cluster.ClusterName.readFrom(ClusterName.java:64)
at
org.elasticsearch.cluster.ClusterName.readClusterName(ClusterName.java:58)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.run(MulticastZenPing.java:409
)
at java.lang.Thread.run(Thread.java:745)
log4j, [2014-09-29T08:36:28.895] WARN:
org.elasticsearch.discovery.zen.ping.multicast: [Asmodeus] failed to
read requesting data from
/172.16.40.29:54328
java.io.IOException: Expected handle header, got [13]
at
org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:65)
at
org.elasticsearch.cluster.ClusterName.readFrom(ClusterName.java:64)
at
org.elasticsearch.cluster.ClusterName.readClusterName(ClusterName.java:58)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.run(MulticastZenPing.java:409
)
at java.lang.Thread.run(Thread.java:745)

{:timestamp=>"2014-09-29T08:36:23.583000-0400", :message=>"Failed to flush
outgoing items", :outgoing_count=>51,
:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException:
waited for [30s],
:backtrace=>["org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180)",
"org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)",
"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)",
"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)",
"java.lang.Thread.run(Thread.java:745)"], :level=>:warn}
{:timestamp=>"2014-09-29T08:36:54.632000-0400", :message=>"Failed to flush
outgoing items", :outgoing_count=>51,
:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException:
waited for [30s],
:backtrace=>["org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180)",
"org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)",
"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)",
"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)",
"java.lang.Thread.run(Thread.java:745)"], :level=>:warn}
{:timestamp=>"2014-09-29T08:37:25.673000-0400", :message=>"Failed to flush
outgoing items", :outgoing_count=>51,
:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException:
waited for [30s],
:backtrace=>["org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180)",
"org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)",
"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)",
"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)",
"java.lang.Thread.run(Thread.java:745)"], :level=>:warn}
{:timestamp=>"2014-09-29T08:37:56.735000-0400", :message=>"Failed to flush
outgoing items", :outgoing_count=>51,
:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException:
waited for [30s],
:backtrace=>["org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180)",
"org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)",
"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)",
"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)",
"java.lang.Thread.run(Thread.java:745)"], :level=>:warn}

Thanks,
Kevin

On Saturday, September 27, 2014 4:34:45 AM UTC-4, Mark Walkom wrote:

Is the cluster external to the LS instance?
Check the logs of your ES master server, it should show the logstash node
join, if you aren't seeing that then enable verbose mode in LS and you
should see the issue.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 27 September 2014 04:28, Kevin M <mcgk...@gmail.com <javascript:>>
wrote:

Do i need to set anything up special to have logstash output to my
elasticsearch cluster?

output {
stdout { }
elasticsearch {
cluster => "es-cluster"
}
}

I have my logstash output setup this way but I do not get anything in
kibana - when I switch to elasticsearch_http it works fine and data shows up

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1a4094e7-2457-4790-9a0e-5cec5bb7b12c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1a4094e7-2457-4790-9a0e-5cec5bb7b12c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e2353e6e-bb64-4f60-bc63-fc5a3d7a6ac3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Think i may have fixed it by upgrading to Logstash 1.4.2

On Monday, September 29, 2014 8:40:32 AM UTC-4, Kevin M wrote:

So here is the really, i think, relevant log message I got:

log4j, [2014-09-29T08:35:22.061] WARN: org.elasticsearch.discovery:
[Asmodeus] waited for 30s and no initial state was set by the discovery

then a bit later:

log4j, [2014-09-29T08:35:22.061] WARN: org.elasticsearch.discovery:
[Asmodeus] waited for 30s and no initial state was set by the discovery

log4j, [2014-09-29T08:36:27.385] WARN:
org.elasticsearch.discovery.zen.ping.multicast: [Asmodeus] failed to
read requesting data from /
172.16.40.29:54328
java.io.IOException: Expected handle header, got [13]
at
org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:65)
at
org.elasticsearch.cluster.ClusterName.readFrom(ClusterName.java:64)
at
org.elasticsearch.cluster.ClusterName.readClusterName(ClusterName.java:58)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.run(MulticastZenPing.java:409
)
at java.lang.Thread.run(Thread.java:745)
log4j, [2014-09-29T08:36:28.895] WARN:
org.elasticsearch.discovery.zen.ping.multicast: [Asmodeus] failed to
read requesting data from /
172.16.40.29:54328
java.io.IOException: Expected handle header, got [13]
at
org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:65)
at
org.elasticsearch.cluster.ClusterName.readFrom(ClusterName.java:64)
at
org.elasticsearch.cluster.ClusterName.readClusterName(ClusterName.java:58)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.run(MulticastZenPing.java:409
)
at java.lang.Thread.run(Thread.java:745)

{:timestamp=>"2014-09-29T08:36:23.583000-0400", :message=>"Failed to flush
outgoing items", :outgoing_count=>51,
:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException:
waited for [30s],
:backtrace=>["org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180)",
"org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)",
"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)",
"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)",
"java.lang.Thread.run(Thread.java:745)"], :level=>:warn}
{:timestamp=>"2014-09-29T08:36:54.632000-0400", :message=>"Failed to flush
outgoing items", :outgoing_count=>51,
:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException:
waited for [30s],
:backtrace=>["org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180)",
"org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)",
"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)",
"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)",
"java.lang.Thread.run(Thread.java:745)"], :level=>:warn}
{:timestamp=>"2014-09-29T08:37:25.673000-0400", :message=>"Failed to flush
outgoing items", :outgoing_count=>51,
:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException:
waited for [30s],
:backtrace=>["org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180)",
"org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)",
"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)",
"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)",
"java.lang.Thread.run(Thread.java:745)"], :level=>:warn}
{:timestamp=>"2014-09-29T08:37:56.735000-0400", :message=>"Failed to flush
outgoing items", :outgoing_count=>51,
:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException:
waited for [30s],
:backtrace=>["org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:180)",
"org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:483)",
"java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)",
"java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)",
"java.lang.Thread.run(Thread.java:745)"], :level=>:warn}

Thanks,
Kevin

On Saturday, September 27, 2014 4:34:45 AM UTC-4, Mark Walkom wrote:

Is the cluster external to the LS instance?
Check the logs of your ES master server, it should show the logstash node
join, if you aren't seeing that then enable verbose mode in LS and you
should see the issue.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 27 September 2014 04:28, Kevin M mcgk...@gmail.com wrote:

Do i need to set anything up special to have logstash output to my
elasticsearch cluster?

output {
stdout { }
elasticsearch {
cluster => "es-cluster"
}
}

I have my logstash output setup this way but I do not get anything in
kibana - when I switch to elasticsearch_http it works fine and data shows up

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1a4094e7-2457-4790-9a0e-5cec5bb7b12c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1a4094e7-2457-4790-9a0e-5cec5bb7b12c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/32e7d5dc-210f-4c53-a5af-254b11ea93fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.