### Author:  [@HeadScratcher](https://discuss.elastic.co/u/HeadScratcher)
#### Post date: [November 19, 2017, 8:40pm UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/1 "2017-11-19T20:40:22Z")
this weekend i've been attempting to get my cluster running.
I chose the Azure market place Elastic/Kibana/X-PACK setup and configured it like this
Single Subnet 10.0.0.0/24
3 data nodes 10.0.0.6, 10.0.0.7, 10.0.0.8
Internal Load Balancer 10.0.0.4
1 Kibana Server 10.0.0.5
When everything had installed, i could access the kibana site via the public address and see the cluster state etc.. but i had no way of pushing data into the 3 data nodes that are running elasticsearch.
From the kibana server i can telnet to ports 9200 and 9300 on the three nodes.
After adding an external IP to each of the three nodes, telnetting to them, i can also see 'back' to the kibana server.
So i install Logstash onto the Kibana server and configure a plugin to pull data from a soure and push it to one of the data nodes... i start getting problems... and just to cover the loadbalancer issue, i tried pushing to that also.
i have three issues i think;
1. can i see in the monitoring page in kibana, the log stash server as part of the cluster or am i not able to do this? or shouldn't ?!
1. i tried pushing data to the ES cluster and got this error 🙂
{:timestamp=\>"2017-11-19T19:58:39.159000+0000", :message=\>"An unexpected error occurred!", :error=\>#\, :class=\>"URI::InvalidURIError", :backtrace=\>["/opt/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:815:in `check_path'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:870:in`path='", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/http\_client.rb:178:in `host_to_url'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/http_client.rb:109:in`build\_client'", "org/jruby/RubyArray.java:2414:in `map'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/http_client.rb:109:in`build\_client'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/http\_client.rb:20:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:40:in`build'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch.rb:132:in `build_client'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/common.rb:14:in`register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/output\_delegator.rb:75:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:173:in`start\_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:173:in`start\_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/pipeline.rb:126:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/agent.rb:210:in`execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:90:in `run'", "org/jruby/RubyProc.java:281:in`call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.4-java/lib/logstash/runner.rb:95:in `run'", "org/jruby/RubyProc.java:281:in`call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:in `initialize'"], :level=\>:warn}
using the following plugin;
input {
twitter {
consumer\_key =\> "REDACTED"
consumer\_secret =\> "REDACTED"
oauth\_token =\> "REDACTED"
oauth\_token\_secret =\> "REDACTED"
keywords =\> ["Call of Duty"]
full\_tweet =\> true
}
}
output {
elasticsearch {
hosts =\> "10.0.0.4" \<---- tried also the other ES IP addresses
bind\_port =\> 9300
index =\> "\*"
cluster =\> "Intel-CL"
document\_type =\> "Twitter"
node\_name =\> 'intelkibana'
}
}
Also i'm not seeing any connections on any of the ES servers running netstat -nlp so its like the Logstash service running on Kibana isn't even connecting, yet i know the ports are open. So should i have a dedicated logstash server? should i not have logstash running on the kibana server?
1. is there any documentation/blogs of anyone who's run the same Azure market place setup ( near enough ) and then added a logstash server into the same subnet and configured it to push data to the ES nodes? as all the research and the blind alleys i've gone down this weekend , two days of chasing links and testing ideas based on Bitnami, [LOGZ.IO](http://LOGZ.IO) and many other examples, i seriously do not understand why there is so much information but none of it is relevant to logstash and clusters... plenty of base data, but nothing on how to pull it all together.
I could instead just create one single large VM in azure, install ELK and have it running in about an hour, but i thought i'd try the clustering approach and use the market place.... im thinking this is a mistake as the market place product should be complete and not half finished as it is, and should have some sort of information on how to push info into it, as on it's own, its pretty useless from my point of view.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 19, 2017, 9:14pm UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/2 "2017-11-19T21:14:27Z")
1. If you install X-Pack in Logstash too, yes.
2. These settings are not valid
> [@HeadScratcher](#):
>
> bind\_port =\> 9300
> index =\> "\*"
> cluster =\> "Intel-CL"
> node\_name =\> 'intelkibana'
You may want to read [Elasticsearch output plugin | Logstash Reference [6.0] | Elastic](https://www.elastic.co/guide/en/logstash/6.0/plugins-outputs-elasticsearch.html)
---
### Author:  [@HeadScratcher](https://discuss.elastic.co/u/HeadScratcher)
#### Post date: [November 20, 2017, 8:43am UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/3 "2017-11-20T08:43:12Z")
so if i create a separate logstash server in the same subnet, then install xpack that will allow it to participate ?
Then kibana server is part of the cluster and has logstash installed on it..or is there a plugin for logstash regarding xpack ?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 20, 2017, 8:55am UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/4 "2017-11-20T08:55:32Z")
You can run Logstash on the same host as Kibana, you just need to install X-Pack as a Logstash plugin to see it in the Monitoring page - [https://www.elastic.co/guide/en/logstash/6.0/setup-xpack.html](https://www.elastic.co/guide/en/logstash/6.0/setup-xpack.html)
---
### Author:  [@HeadScratcher](https://discuss.elastic.co/u/HeadScratcher)
#### Post date: [November 20, 2017, 8:56am UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/5 "2017-11-20T08:56:26Z")
ahhhh
thanks 🙂
---
### Author:  [@HeadScratcher](https://discuss.elastic.co/u/HeadScratcher)
#### Post date: [November 20, 2017, 10:45am UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/6 "2017-11-20T10:45:43Z")
So i looked at [https://www.elastic.co/guide/en/logstash/6.0/setup-xpack.html](https://www.elastic.co/guide/en/logstash/6.0/setup-xpack.html) and followed the instructions..
then i ran this to install direct from the internet....
( notice all documentation says to use "logstash-plugin" instead of "plugin" which isn't the case in my 2.2.4 install as i have no logstash-plugin file to run)
elkman@Intelkibana:/opt/logstash/bin$ sudo ./plugin install x-pack
Validating x-pack
x-pack is not a Logstash plugin
ERROR: Installation aborted, verification failed for x-pack
the logstash plugin its basically one .gem file, which i extracted and put into /opt/logstash dir then ran this command...
sudo ./plugin install /opt/logstash/x-pack-6.0.0-java.gem
i get this message:
elkman@Intelkibana:/opt/logstash/bin$ sudo ./plugin install /opt/logstash/x-pack-6.0.0-java.gem
Validating /opt/logstash/x-pack-6.0.0-java.gem
Installing x-pack
Error Bundler::GemspecError, retrying 1/10
There was a Errno::ENOENT while loading x-pack.gemspec:
No such file or directory - /opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from
/opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in `eval_gemspec' Error Bundler::GemspecError, retrying 2/10 There was a Errno::ENOENT while loading x-pack.gemspec: No such file or directory - /opt/logstash/vendor/local_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from /opt/logstash/vendor/local_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in`eval\_gemspec'
Error Bundler::GemspecError, retrying 3/10
There was a Errno::ENOENT while loading x-pack.gemspec:
No such file or directory - /opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from
/opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in `eval_gemspec' Error Bundler::GemspecError, retrying 4/10 There was a Errno::ENOENT while loading x-pack.gemspec: No such file or directory - /opt/logstash/vendor/local_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from /opt/logstash/vendor/local_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in`eval\_gemspec'
Error Bundler::GemspecError, retrying 5/10
There was a Errno::ENOENT while loading x-pack.gemspec:
No such file or directory - /opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from
/opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in `eval_gemspec' Error Bundler::GemspecError, retrying 6/10 There was a Errno::ENOENT while loading x-pack.gemspec: No such file or directory - /opt/logstash/vendor/local_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from /opt/logstash/vendor/local_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in`eval\_gemspec'
Error Bundler::GemspecError, retrying 7/10
There was a Errno::ENOENT while loading x-pack.gemspec:
No such file or directory - /opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from
/opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in `eval_gemspec' Error Bundler::GemspecError, retrying 8/10 There was a Errno::ENOENT while loading x-pack.gemspec: No such file or directory - /opt/logstash/vendor/local_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from /opt/logstash/vendor/local_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in`eval\_gemspec'
Error Bundler::GemspecError, retrying 9/10
There was a Errno::ENOENT while loading x-pack.gemspec:
No such file or directory - /opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from
/opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in `eval_gemspec' Error Bundler::GemspecError, retrying 10/10 There was a Errno::ENOENT while loading x-pack.gemspec: No such file or directory - /opt/logstash/vendor/local_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from /opt/logstash/vendor/local_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in`eval\_gemspec'
Too many retries, aborting, caused by Bundler::GemspecError
ERROR: Installation Aborted, message: There was a Errno::ENOENT while loading x-pack.gemspec:
No such file or directory - /opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/VERSION from
/opt/logstash/vendor/local\_gems/0d15e0fe/x-pack-6.0.0-java/x-pack.gemspec:3:in `eval\_gemspec'
what am i doing wrong ? i thought you could pull down the plugins from a repo of some kind and it will install
---
### Author:  [@HeadScratcher](https://discuss.elastic.co/u/HeadScratcher)
#### Post date: [November 20, 2017, 2:54pm UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/7 "2017-11-20T14:54:59Z")
scouring the internet for information, everything seems to point towards installing x-pack into an ES instance then installing the x-pack plugin into logstash that is sitting on the same machine as ES, but nothing about installing logstash x-pack plugin as a stand alone.
As the market place install of my 3 node cluster and 1 kibana vm already has x-pack installed, i opted to install logstash on the kibana vm and try to get the x-pack logstash plugin installed.... no such luck.
Anyone come across running or installing the x-pack plugin into logstash before ?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 20, 2017, 7:03pm UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/8 "2017-11-20T19:03:54Z")
X-Pack for Logstash is not available for 2.X, is there a reason you cannot upgrade?
---
### Author:  [@przemolb](https://discuss.elastic.co/u/przemolb)
#### Post date: [November 21, 2017, 9:50am UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/9 "2017-11-21T09:50:36Z")
> [@warkolm](#):
>
> You can run Logstash on the same host as Kibana, you just need to install X-Pack as a Logstash plugin to see it in the Monitoring page - [Setting Up X-Pack | Logstash Reference [6.0] | Elastic](https://www.elastic.co/guide/en/logstash/6.0/setup-xpack.html)
Can I install Logstash on Kibana host without using X-Pack ?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 21, 2017, 10:02am UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/10 "2017-11-21T10:02:55Z")
Yes but you won't see Logstash in Monitoring.
---
### Author:  [@przemolb](https://discuss.elastic.co/u/przemolb)
#### Post date: [November 21, 2017, 11:09am UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/11 "2017-11-21T11:09:22Z")
Hmm, AFAIR X-Pack is commercial. So we have to buy it, right ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 21, 2017, 11:12am UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/12 "2017-11-21T11:12:59Z")
X-Pack is commercial, but there is a [Basic License](https://www.elastic.co/subscriptions) that is free and gives access to Monitoring. You just need to register for it.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 19, 2017, 11:13am UTC](https://discuss.elastic.co/t/new-logstash-server-in-azure-elastic-3-node-cluster/108297/13 "2017-12-19T11:13:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.