# Question about Logstash Joining ES Cluster and Index **URL:** \ **Category:** Elasticsearch\ **Created:** [November 14, 2014, 9:26pm UTC](https://discuss.elastic.co/t/question-about-logstash-joining-es-cluster-and-index/20760 "2014-11-14T21:26:22Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![SQLBigG](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@SQLBigG](https://discuss.elastic.co/u/SQLBigG)\ **Post date:** [November 14, 2014, 9:26pm UTC](https://discuss.elastic.co/t/question-about-logstash-joining-es-cluster-and-index/20760/1 "2014-11-14T21:26:22Z")
My setup: Logstash Node parsing NXLog eventlogs from windows servers and sending them to a 3 node ES Cluster. Here's my logstash conf file: input { tcp { type =\> "eventlog" host =\> "logstash01" port =\> 3515 codec =\> 'json' } } output { elasticsearch { index =\> "logstash-events-np-%{+YYYY-MM-dd}" } } Here is the elasticsearch.yml file on my logstash server: cluster.name: ELCluster node.name: logstash01 discovery.zen.ping.unicasthosts: ["elsearch01", "elsearch02"] (master nodes) So when I look at HEAD, I can see that the logstash node has joined the cluster successfully, but my index is not being built. In the logstash log, I see the following: log4j, [2014-11-14T13:22:49.842] WARN: org.elasticsearch.discovery: [logstash-logstash01-29863-2036] waited for 30s and no initial state was set by the discovery Not sure why it's not building my index as I know there is data coming in from about 50 servers or so. I'm a little lost at this point as I was able to get this far and don't know if I am missing a setting somewhere. All servers are on the same network segment and there are no firewalls on the servers or between the servers. I was able to build basic indexes without an issue from the ES cluster using the basic commands so I know it has the ability to create an index. Originally, the current logstash server and elasticsearch server were our PoC servers and were successful and now we wanted to build a cluster for ES since it was going to be handling a large amount of data. Thanks in advance. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8166101a-4052-4ef0-be30-672aaefaa85d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8166101a-4052-4ef0-be30-672aaefaa85d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [November 14, 2014, 10:34pm UTC](https://discuss.elastic.co/t/question-about-logstash-joining-es-cluster-and-index/20760/2 "2014-11-14T22:34:46Z")
I highly recommend that you use the HTTP output. Works great, is immune to the ES version, and there are no performance issues that I've seen. It Just Works. For example, here's my sample logstash configuration's output settings: output { # Uncomment for testing only: # stdout { codec =\> rubydebug } # Elasticsearch elasticsearch { # Specify http (with or without quotes around http) to direct the # output as JSON documents via the Elasticsearch HTTP REST API protocol =\> "http" codec =\> json manage\_template =\> false ``` # Or whatever target ES host is required host => "localhost" # Or whatever _type is desired: index_type => "sample" ``` } } As you can probably surmise, I have my own default index creation template so there's no need to splatter it all over creation; logstash runs better on the host on which it's gathering the log files and I vastly prefer one central index template than keeping a bazillion logstash configurations in perfect sync. And if we happen replace logstash for something else, then I still have my index creation templates. Hope this helps! Brian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/27854489-1f4d-4ebd-883c-64dc6235eed4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/27854489-1f4d-4ebd-883c-64dc6235eed4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![SQLBigG](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@SQLBigG](https://discuss.elastic.co/u/SQLBigG)\ **Post date:** [November 17, 2014, 2:28pm UTC](https://discuss.elastic.co/t/question-about-logstash-joining-es-cluster-and-index/20760/3 "2014-11-17T14:28:13Z")
Brian, I've tried using the http protocol and it did not work. The Cluster sees the logstash node, but it's not creating the index still. Thanks, Gerald On Friday, November 14, 2014 4:26:22 PM UTC-5, @SQLBigG wrote: > My setup: > > Logstash Node parsing NXLog eventlogs from windows servers and sending > them to a 3 node ES Cluster. > > Here's my logstash conf file: > > input { > tcp { > type =\> "eventlog" > host =\> "logstash01" > port =\> 3515 > codec =\> 'json' > } > } > > output { > elasticsearch > { > index =\> "logstash-events-np-%{+YYYY-MM-dd}" > } > } > > Here is the elasticsearch.yml file on my logstash server: > > cluster.name: ELCluster > node.name: logstash01 > discovery.zen.ping.unicasthosts: ["elsearch01", "elsearch02"] (master > nodes) > > So when I look at HEAD, I can see that the logstash node has joined the > cluster successfully, but my index is not being built. In the logstash > log, I see the following: > > log4j, [2014-11-14T13:22:49.842] WARN: org.elasticsearch.discovery: > [logstash-logstash01-29863-2036] waited for 30s and no initial state was > set by the discovery > > Not sure why it's not building my index as I know there is data coming in > from about 50 servers or so. I'm a little lost at this point as I was able > to get this far and don't know if I am missing a setting somewhere. All > servers are on the same network segment and there are no firewalls on the > servers or between the servers. I was able to build basic indexes without > an issue from the ES cluster using the basic commands so I know it has the > ability to create an index. > > Originally, the current logstash server and elasticsearch server were our > PoC servers and were successful and now we wanted to build a cluster for ES > since it was going to be handling a large amount of data. > > Thanks in advance. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1d852a5d-8621-4b76-b127-c7a886c00561%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1d852a5d-8621-4b76-b127-c7a886c00561%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:49am UTC](https://discuss.elastic.co/t/question-about-logstash-joining-es-cluster-and-index/20760/4 "2017-07-06T00:49:32Z")