Question about Logstash Joining ES Cluster and Index

Elastic Stack Elasticsearch
1

My setup:

Logstash Node parsing NXLog eventlogs from windows servers and sending them
to a 3 node ES Cluster.

Here's my logstash conf file:

input {
tcp {
type => "eventlog"
host => "logstash01"
port => 3515
codec => 'json'
}
}

output {
elasticsearch
{
index => "logstash-events-np-%{+YYYY-MM-dd}"
}
}

Here is the elasticsearch.yml file on my logstash server:

cluster.name: ELCluster
node.name: logstash01
discovery.zen.ping.unicasthosts: ["elsearch01", "elsearch02"] (master nodes)

So when I look at HEAD, I can see that the logstash node has joined the
cluster successfully, but my index is not being built. In the logstash
log, I see the following:

log4j, [2014-11-14T13:22:49.842] WARN: org.elasticsearch.discovery:
[logstash-logstash01-29863-2036] waited for 30s and no initial state was
set by the discovery

Not sure why it's not building my index as I know there is data coming in
from about 50 servers or so. I'm a little lost at this point as I was able
to get this far and don't know if I am missing a setting somewhere. All
servers are on the same network segment and there are no firewalls on the
servers or between the servers. I was able to build basic indexes without
an issue from the ES cluster using the basic commands so I know it has the
ability to create an index.

Originally, the current logstash server and elasticsearch server were our
PoC servers and were successful and now we wanted to build a cluster for ES
since it was going to be handling a large amount of data.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8166101a-4052-4ef0-be30-672aaefaa85d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I highly recommend that you use the HTTP output. Works great, is immune to
the ES version, and there are no performance issues that I've seen. It Just
Works.

For example, here's my sample logstash configuration's output settings:

output {

Uncomment for testing only:

stdout { codec => rubydebug }

Elasticsearch

elasticsearch {
# Specify http (with or without quotes around http) to direct the
# output as JSON documents via the Elasticsearch HTTP REST API
protocol => "http"
codec => json
manage_template => false

 # Or whatever target ES host is required
 host => "localhost"

 # Or whatever _type is desired:
 index_type => "sample"

}
}

As you can probably surmise, I have my own default index creation template
so there's no need to splatter it all over creation; logstash runs better
on the host on which it's gathering the log files and I vastly prefer one
central index template than keeping a bazillion logstash configurations in
perfect sync. And if we happen replace logstash for something else, then I
still have my index creation templates.

Hope this helps!

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/27854489-1f4d-4ebd-883c-64dc6235eed4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Brian,

I've tried using the http protocol and it did not work. The Cluster sees
the logstash node, but it's not creating the index still.

Thanks,
Gerald

On Friday, November 14, 2014 4:26:22 PM UTC-5, @SQLBigG wrote:

My setup:

Logstash Node parsing NXLog eventlogs from windows servers and sending
them to a 3 node ES Cluster.

Here's my logstash conf file:

input {
tcp {
type => "eventlog"
host => "logstash01"
port => 3515
codec => 'json'
}
}

output {
elasticsearch
{
index => "logstash-events-np-%{+YYYY-MM-dd}"
}
}

Here is the elasticsearch.yml file on my logstash server:

cluster.name: ELCluster
node.name: logstash01
discovery.zen.ping.unicasthosts: ["elsearch01", "elsearch02"] (master
nodes)

So when I look at HEAD, I can see that the logstash node has joined the
cluster successfully, but my index is not being built. In the logstash
log, I see the following:

log4j, [2014-11-14T13:22:49.842] WARN: org.elasticsearch.discovery:
[logstash-logstash01-29863-2036] waited for 30s and no initial state was
set by the discovery

Not sure why it's not building my index as I know there is data coming in
from about 50 servers or so. I'm a little lost at this point as I was able
to get this far and don't know if I am missing a setting somewhere. All
servers are on the same network segment and there are no firewalls on the
servers or between the servers. I was able to build basic indexes without
an issue from the ES cluster using the basic commands so I know it has the
ability to create an index.

Originally, the current logstash server and elasticsearch server were our
PoC servers and were successful and now we wanted to build a cluster for ES
since it was going to be handling a large amount of data.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1d852a5d-8621-4b76-b127-c7a886c00561%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.