# Unable to detect new logstash instances
**URL:** https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085
**Category:** Logstash
**Created:** [March 21, 2018, 11:37pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085 "2018-03-21T23:37:10Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [March 21, 2018, 11:37pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/1 "2018-03-21T23:37:10Z")
Hi ,
i have ELK stack running and X-pack monitoring is able to detect first 3 logstash node and able to monitor . But i have added 3 more logstash nodes are not able to detect by x-pack monitoring .
```
[2018-03-21T23:33:31,244][INFO][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.8.12:9200/, :path=>"/"}
[2018-03-21T23:33:33,194][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.4.12:9200/, :path=>"/"}
[2018-03-21T23:33:34,248][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.4.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable:
[https://logstash_system:xxxxxx@10.62.4.12:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2018-03-21T23:33:34,248][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.8.12:9200/, :path=>"/"}
[2018-03-21T23:33:41,273][WARN][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.8.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.8.12:9200/][Manticore::ConnectTimeout] connect timed out"}
[2018-03-21T23:33:41,273][INFO][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.12.12:9200/, :path=>"/"}
[2018-03-21T23:33:44,255][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.8.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.8.12:9200/][Manticore::ConnectTimeout] connect timed out"}
[2018-03-21T23:33:44,255][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.12.12:9200/, :path=>"/"}
[2018-03-21T23:33:51,279][WARN][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.12.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.12.12:9200/][Manticore::ConnectTimeout] connect timed out"}
[2018-03-21T23:33:54,295][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.12.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.12.12:9200/][Manticore::ConnectTimeout] connect timed out"}
[2018-03-21T23:33:55,296][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.4.12:9200/, :path=>"/"}
```
I have already updated logstash.yml
```
more logstash.yml | grep -v "#"
path.data: /var/lib/logstash
path.logs: /var/log/logstash
node.name: iacapps-logstash-node4
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 'PW'
xpack.monitoring.elasticsearch.url: ["https://x.x.x.x:9200", "https://x.x.x.x:9200", "https://x.x.x.x:9200"]
xpack.monitoring.elasticsearch.ssl.ca: /etc/logstash/certs/ca.crt
```
can someone help
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 22, 2018, 5:30am UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/2 "2018-03-22T05:30:29Z")
> [@Nikhilpawar1985](#):
>
> No route to host (Host unreachable)
That would suggest that there is a networking issue somewhere.
Can you ping/telnet between the various hosts?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2018, 7:27am UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/3 "2018-03-22T07:27:33Z")
Did you install the first 3 Logstash instances differently compared to the last 3 ones? I recall seeing this issue if the installation has simply been copied so that several Logstash instances have the same ID.
---
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [March 22, 2018, 1:46pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/4 "2018-03-22T13:46:10Z")
My logstash and ES hosts are on AWS private subnets . The first 3 and next 3 are. I have distributed them like
logstash1 & logstash 4 , es-node1 \>\>\>\>\>subnet1
logstash2 & logstash 5 , es-node2 \>\>\>\>\>subnet2
logstash3 & logstash 6 , es-node3 \>\>\>\>\>subnet3
They all are having same versions
```
[root@iacapps-logstash-node1 ~]# /usr/share/logstash/bin/logstash --version
logstash 6.2.3
[root@iacapps-logstash-node4 ~]# /usr/share/logstash/bin/logstash --version
logstash 6.2.3
```
There should not be any connectivity issues as everything is same on network side .
I wounder its something on ES license ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2018, 2:16pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/5 "2018-03-22T14:16:43Z")
How did you install Logstash on the hosts? Are they all configured the same way?
---
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [March 22, 2018, 2:21pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/6 "2018-03-22T14:21:07Z")
Yes . I have red-hat OS
They are configured exactly same .
yum rpm package
I am able to see that its processing logs but Kibana x-pack monitoring not showing them only first 3 nodes
```
[root@iacapps-logstash-node1 ~]# curl -XGET 'localhost:9600/_node/stats/events?pretty'
{
"host" : "iacapps-logstash-node1",
"version" : "6.2.3",
"http_address" : "127.0.0.1:9600",
"id" : "75f0b76a-1d7a-42bb-9847-082643b47b7f",
"name" : "iacapps-logstash-node1",
"events" : {
"in" : 17719359,
"filtered" : 17719256,
"out" : 17719256,
"duration_in_millis" : 43537761,
"queue_push_duration_in_millis" : 6221387
}
[root@iacapps-logstash-node4 ~]# curl -XGET 'localhost:9600/_node/stats/events?pretty'
{
"host" : "iacapps-logstash-node4",
"version" : "6.2.3",
"http_address" : "127.0.0.1:9600",
"id" : "fe7c21d5-d195-4ea8-9d1c-7e84ae0582bc",
"name" : "iacapps-logstash-node4",
"events" : {
"in" : 1201,
"filtered" : 1201,
"out" : 1201,
"duration_in_millis" : 15745,
"queue_push_duration_in_millis" : 0
}
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2018, 2:26pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/7 "2018-03-22T14:26:47Z")
Can you show the stats output for the other nodes as well? Any differences in the `logstash.yml` config between the nodes that do and do not show up?
---
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [March 22, 2018, 2:31pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/8 "2018-03-22T14:31:39Z")
Other nodes stats are also good
```
[root@iacapps-logstash-node2 ~]# curl -XGET 'localhost:9600/_node/stats/events?pretty'
{
"host" : "iacapps-logstash-node2",
"version" : "6.2.3",
"http_address" : "127.0.0.1:9600",
"id" : "a682b327-0eca-4538-8f4c-45f58ec3490b",
"name" : "iacapps-logstash-node2",
"events" : {
"in" : 17882553,
"filtered" : 17882530,
"out" : 17882530,
"duration_in_millis" : 41877720,
"queue_push_duration_in_millis" : 6042907
}
[root@iacapps-logstash-node1 ~]# more /etc/logstash/logstash.yml | grep -v "#"
path.data: /var/lib/logstash
path.logs: /var/log/logstash
node.name: iacapps-logstash-node1
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 'PW'
xpack.monitoring.elasticsearch.url: ["https://10.62.4.15:9200", "https://10.62.8.15:9200", "https://10.62.12.15:9200"]
xpack.monitoring.elasticsearch.ssl.ca: /etc/logstash/certs/ca.crt
[root@iacapps-logstash-node4 ~]# more /etc/logstash/logstash.yml | grep -v "#"
path.data: /var/lib/logstash
path.logs: /var/log/logstash
node.name: iacapps-logstash-node4
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 'PW'
xpack.monitoring.elasticsearch.url: ["https://10.62.4.12:9200", "https://10.62.8.12:9200", "https://10.62.12.12:9200"]
xpack.monitoring.elasticsearch.ssl.ca: /etc/logstash/certs/ca.crt
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2018, 2:43pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/9 "2018-03-22T14:43:34Z")
Anything in the Logstash logs? If not, can you increase the logging level to `debug`?
---
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [March 22, 2018, 2:51pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/10 "2018-03-22T14:51:19Z")
Hi ,
Enabled debug logging and restarted logstash node
Seeing this in logs
```
[io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: SSL_RSA_WITH_3DES_EDE_CBC_SHA => DES-CBC3-SHA
[2018-03-22T14:46:40,396][DEBUG][io.netty.handler.ssl.OpenSsl] Supported protocols (OpenSSL): [[SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2]]
[2018-03-22T14:46:40,396][DEBUG][io.netty.handler.ssl.OpenSsl] Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA]
[2018-03-22T14:46:40,397][INFO][logstash.inputs.beats] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-03-22T14:46:40,422][DEBUG][io.netty.channel.MultithreadEventLoopGroup] -Dio.netty.eventLoopThreads: 4
[2018-03-22T14:46:40,453][DEBUG][io.netty.channel.nio.NioEventLoop] -Dio.netty.noKeySetOptimization: false
[2018-03-22T14:46:40,453][DEBUG][io.netty.channel.nio.NioEventLoop] -Dio.netty.selectorAutoRebuildThreshold: 512
[2018-03-22T14:46:40,475][DEBUG][io.netty.util.internal.PlatformDependent] org.jctools-core.MpscChunkedArrayQueue: available
[2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
[2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
[2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
[2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
[2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
[2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
[2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
[2018-03-22T14:46:40,505][INFO][org.logstash.beats.Server] Starting server on port: 5044
[2018-03-22T14:46:40,525][INFO][logstash.pipeline] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#"}
[2018-03-22T14:46:40,529][INFO][logstash.agent] Pipelines running {:count=>2, :pipelines=>[".monitoring-logstash", "main"]}
[2018-03-22T14:46:40,530][ERROR][logstash.inputs.metrics] Monitoring is not available: License information is currently unavailable. Please make sure you have added your production elasticsearch connection info in the xpack.monitoring.elasticsearch settings.
[2018-03-22T14:46:40,541][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-03-22T14:46:40,541][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-03-22T14:46:40,629][DEBUG][io.netty.channel.DefaultChannelId] -Dio.netty.processId: 14692 (auto-detected)
[2018-03-22T14:46:40,631][DEBUG][io.netty.util.NetUtil] -Djava.net.preferIPv4Stack: true
[2018-03-22T14:46:40,631][DEBUG][io.netty.util.NetUtil] -Djava.net.preferIPv6Addresses: false
[2018-03-22T14:46:40,633][DEBUG][io.netty.util.NetUtil] Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
[2018-03-22T14:46:40,634][DEBUG][io.netty.util.NetUtil] /proc/sys/net/core/somaxconn: 128
[2018-03-22T14:46:40,635][DEBUG][io.netty.channel.DefaultChannelId] -Dio.netty.machineId: 0a:4c:2c:ff:fe:24:bb:0a (auto-detected)
[2018-03-22T14:46:41,092][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.4.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.4.12:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2018-03-22T14:46:41,092][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.8.12:9200/, :path=>"/"}
[2018-03-22T14:46:41,099][WARN][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.4.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.4.12:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
[2018-03-22T14:46:41,099][INFO][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.8.12:9200/, :path=>"/"}
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 22, 2018, 3:19pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/11 "2018-03-22T15:19:38Z")
> [@Nikhilpawar1985](#):
>
> [2018-03-22T14:46:41,092][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"[https://logstash\_system:xxxxxx@10.62.4.12:9200/](https://logstash_system:xxxxxx@10.62.4.12:9200/)", :error\_type=\>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=\>"Elasticsearch Unreachable: [[https://logstash\_system:xxxxxx@10.62.4.12:9200/](https://logstash_system:xxxxxx@10.62.4.12:9200/)][Manticore::SocketException] No route to host (Host unreachable)"}
> [2018-03-22T14:46:41,092][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[https://logstash\_system:xxxxxx@10.62.8.12:9200/](https://logstash_system:xxxxxx@10.62.8.12:9200/), :path=\>"/"}
> [2018-03-22T14:46:41,099][WARN][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"[https://logstash\_system:xxxxxx@10.62.4.12:9200/](https://logstash_system:xxxxxx@10.62.4.12:9200/)", :error\_type=\>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=\>"Elasticsearch Unreachable: [[https://logstash\_system:xxxxxx@10.62.4.12:9200/](https://logstash_system:xxxxxx@10.62.4.12:9200/)][Manticore::SocketException] No route to host (Host unreachable)"}
> [2018-03-22T14:46:41,099][INFO][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[https://logstash\_system:xxxxxx@10.62.8.12:9200/](https://logstash_system:xxxxxx@10.62.8.12:9200/), :path=\>"/"}
Are you sure there are no network or firewall issues?
---
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [March 22, 2018, 3:36pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/12 "2018-03-22T15:36:23Z")
i dont think
I am able to reach ES nodes , check below
```
[root@iacapps-logstash-node1 ~]# curl --cacert /etc/logstash/certs/ca.crt -u logstash_system https://10.62.4.15:9200
Enter host password for user 'logstash_system':
{
"name" : "iacapps-es-node1",
"cluster_name" : "iacapps-elastic",
"cluster_uuid" : "KyJpUNUxR3a0Fl5HOF0-qQ",
"version" : {
"number" : "6.2.3",
"build_hash" : "c59ff00",
"build_date" : "2018-03-13T10:06:29.741383Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
[root@iacapps-logstash-node4 ~]# curl --cacert /etc/logstash/certs/ca.crt -u logstash_system https://10.62.4.15:9200
Enter host password for user 'logstash_system':
{
"name" : "iacapps-es-node1",
"cluster_name" : "iacapps-elastic",
"cluster_uuid" : "KyJpUNUxR3a0Fl5HOF0-qQ",
"version" : {
"number" : "6.2.3",
"build_hash" : "c59ff00",
"build_date" : "2018-03-13T10:06:29.741383Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
```
---
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [March 22, 2018, 8:29pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/13 "2018-03-22T20:29:17Z")
Reset the password , its working now
Not sure if that was the issue
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 19, 2018, 8:29pm UTC](https://discuss.elastic.co/t/unable-to-detect-new-logstash-instances/125085/14 "2018-04-19T20:29:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.