Unable to detect new logstash instances

Elastic Stack Logstash
1

Hi ,

i have ELK stack running and X-pack monitoring is able to detect first 3 logstash node and able to monitor . But i have added 3 more logstash nodes are not able to detect by x-pack monitoring .

[2018-03-21T23:33:31,244][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.8.12:9200/, :path=>"/"}
[2018-03-21T23:33:33,194][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.4.12:9200/, :path=>"/"}
[2018-03-21T23:33:34,248][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.4.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable:     
[https://logstash_system:xxxxxx@10.62.4.12:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
    [2018-03-21T23:33:34,248][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.8.12:9200/, :path=>"/"}
    [2018-03-21T23:33:41,273][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.8.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.8.12:9200/][Manticore::ConnectTimeout] connect timed out"}
    [2018-03-21T23:33:41,273][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.12.12:9200/, :path=>"/"}
    [2018-03-21T23:33:44,255][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.8.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.8.12:9200/][Manticore::ConnectTimeout] connect timed out"}
    [2018-03-21T23:33:44,255][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.12.12:9200/, :path=>"/"}
    [2018-03-21T23:33:51,279][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.12.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.12.12:9200/][Manticore::ConnectTimeout] connect timed out"}
    [2018-03-21T23:33:54,295][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.12.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.12.12:9200/][Manticore::ConnectTimeout] connect timed out"}
    [2018-03-21T23:33:55,296][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.4.12:9200/, :path=>"/"}

I have already updated logstash.yml

 more logstash.yml | grep -v "#"
path.data: /var/lib/logstash
path.logs: /var/log/logstash
node.name:  iacapps-logstash-node4
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 'PW'
xpack.monitoring.elasticsearch.url: ["https://x.x.x.x:9200", "https://x.x.x.x:9200", "https://x.x.x.x:9200"]
xpack.monitoring.elasticsearch.ssl.ca: /etc/logstash/certs/ca.crt

can someone help

2

That would suggest that there is a networking issue somewhere.

Can you ping/telnet between the various hosts?

3

Did you install the first 3 Logstash instances differently compared to the last 3 ones? I recall seeing this issue if the installation has simply been copied so that several Logstash instances have the same ID.

4

My logstash and ES hosts are on AWS private subnets . The first 3 and next 3 are. I have distributed them like
logstash1 & logstash 4 , es-node1 >>>>>subnet1
logstash2 & logstash 5 , es-node2 >>>>>subnet2
logstash3 & logstash 6 , es-node3 >>>>>subnet3

They all are having same versions

[root@iacapps-logstash-node1 ~]# /usr/share/logstash/bin/logstash --version
logstash 6.2.3
[root@iacapps-logstash-node4 ~]# /usr/share/logstash/bin/logstash --version
logstash 6.2.3

There should not be any connectivity issues as everything is same on network side .
I wounder its something on ES license ?

5

How did you install Logstash on the hosts? Are they all configured the same way?

6

Yes . I have red-hat OS
They are configured exactly same .

yum rpm package
I am able to see that its processing logs but Kibana x-pack monitoring not showing them only first 3 nodes

[root@iacapps-logstash-node1 ~]# curl -XGET 'localhost:9600/_node/stats/events?pretty'
{
  "host" : "iacapps-logstash-node1",
  "version" : "6.2.3",
  "http_address" : "127.0.0.1:9600",
  "id" : "75f0b76a-1d7a-42bb-9847-082643b47b7f",
  "name" : "iacapps-logstash-node1",
  "events" : {
    "in" : 17719359,
    "filtered" : 17719256,
    "out" : 17719256,
    "duration_in_millis" : 43537761,
    "queue_push_duration_in_millis" : 6221387
  }

[root@iacapps-logstash-node4 ~]# curl -XGET 'localhost:9600/_node/stats/events?pretty'
{
  "host" : "iacapps-logstash-node4",
  "version" : "6.2.3",
  "http_address" : "127.0.0.1:9600",
  "id" : "fe7c21d5-d195-4ea8-9d1c-7e84ae0582bc",
  "name" : "iacapps-logstash-node4",
  "events" : {
    "in" : 1201,
    "filtered" : 1201,
    "out" : 1201,
    "duration_in_millis" : 15745,
    "queue_push_duration_in_millis" : 0
  }

image
image.png880×222 9.91 KB

image
image.png1081×392 37.6 KB

7

Can you show the stats output for the other nodes as well? Any differences in the logstash.yml config between the nodes that do and do not show up?

8

Other nodes stats are also good

[root@iacapps-logstash-node2 ~]# curl -XGET 'localhost:9600/_node/stats/events?pretty'
{
  "host" : "iacapps-logstash-node2",
  "version" : "6.2.3",
  "http_address" : "127.0.0.1:9600",
  "id" : "a682b327-0eca-4538-8f4c-45f58ec3490b",
  "name" : "iacapps-logstash-node2",
  "events" : {
    "in" : 17882553,
    "filtered" : 17882530,
    "out" : 17882530,
    "duration_in_millis" : 41877720,
    "queue_push_duration_in_millis" : 6042907
  }

[root@iacapps-logstash-node1 ~]# more /etc/logstash/logstash.yml | grep -v "#"
path.data: /var/lib/logstash
path.logs: /var/log/logstash
node.name:  iacapps-logstash-node1
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 'PW'
xpack.monitoring.elasticsearch.url: ["https://10.62.4.15:9200", "https://10.62.8.15:9200", "https://10.62.12.15:9200"]
xpack.monitoring.elasticsearch.ssl.ca: /etc/logstash/certs/ca.crt


[root@iacapps-logstash-node4 ~]# more /etc/logstash/logstash.yml | grep -v "#"
path.data: /var/lib/logstash
path.logs: /var/log/logstash
node.name:  iacapps-logstash-node4
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 'PW'
xpack.monitoring.elasticsearch.url: ["https://10.62.4.12:9200", "https://10.62.8.12:9200", "https://10.62.12.12:9200"]
xpack.monitoring.elasticsearch.ssl.ca: /etc/logstash/certs/ca.crt
9

Anything in the Logstash logs? If not, can you increase the logging level to debug?

10

Hi ,
Enabled debug logging and restarted logstash node

Seeing this in logs

[io.netty.handler.ssl.CipherSuiteConverter] Cipher suite mapping: SSL_RSA_WITH_3DES_EDE_CBC_SHA => DES-CBC3-SHA
            [2018-03-22T14:46:40,396][DEBUG][io.netty.handler.ssl.OpenSsl] Supported protocols (OpenSSL): [[SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2]]
            [2018-03-22T14:46:40,396][DEBUG][io.netty.handler.ssl.OpenSsl] Default cipher suites (OpenSSL): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA]
            [2018-03-22T14:46:40,397][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
            [2018-03-22T14:46:40,422][DEBUG][io.netty.channel.MultithreadEventLoopGroup] -Dio.netty.eventLoopThreads: 4
            [2018-03-22T14:46:40,453][DEBUG][io.netty.channel.nio.NioEventLoop] -Dio.netty.noKeySetOptimization: false
            [2018-03-22T14:46:40,453][DEBUG][io.netty.channel.nio.NioEventLoop] -Dio.netty.selectorAutoRebuildThreshold: 512
            [2018-03-22T14:46:40,475][DEBUG][io.netty.util.internal.PlatformDependent] org.jctools-core.MpscChunkedArrayQueue: available
            [2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
            [2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
            [2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
            [2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
            [2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
            [2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
            [2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
            [2018-03-22T14:46:40,480][DEBUG][org.logstash.netty.SslSimpleBuilder] Cipher is supported: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
            [2018-03-22T14:46:40,505][INFO ][org.logstash.beats.Server] Starting server on port: 5044
            [2018-03-22T14:46:40,525][INFO ][logstash.pipeline        ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2a71db0b@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run>"}
            [2018-03-22T14:46:40,529][INFO ][logstash.agent           ] Pipelines running {:count=>2, :pipelines=>[".monitoring-logstash", "main"]}
            [2018-03-22T14:46:40,530][ERROR][logstash.inputs.metrics  ] Monitoring is not available: License information is currently unavailable. Please make sure you have added your production elasticsearch connection info in the xpack.monitoring.elasticsearch settings.
            [2018-03-22T14:46:40,541][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
            [2018-03-22T14:46:40,541][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
            [2018-03-22T14:46:40,629][DEBUG][io.netty.channel.DefaultChannelId] -Dio.netty.processId: 14692 (auto-detected)
            [2018-03-22T14:46:40,631][DEBUG][io.netty.util.NetUtil    ] -Djava.net.preferIPv4Stack: true
            [2018-03-22T14:46:40,631][DEBUG][io.netty.util.NetUtil    ] -Djava.net.preferIPv6Addresses: false
            [2018-03-22T14:46:40,633][DEBUG][io.netty.util.NetUtil    ] Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
            [2018-03-22T14:46:40,634][DEBUG][io.netty.util.NetUtil    ] /proc/sys/net/core/somaxconn: 128
            [2018-03-22T14:46:40,635][DEBUG][io.netty.channel.DefaultChannelId] -Dio.netty.machineId: 0a:4c:2c:ff:fe:24:bb:0a (auto-detected)
            [2018-03-22T14:46:41,092][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.4.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.4.12:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
            [2018-03-22T14:46:41,092][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.8.12:9200/, :path=>"/"}
            [2018-03-22T14:46:41,099][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@10.62.4.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.62.4.12:9200/][Manticore::SocketException] No route to host (Host unreachable)"}
            [2018-03-22T14:46:41,099][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@10.62.8.12:9200/, :path=>"/"}
11

Are you sure there are no network or firewall issues?

12

i dont think

I am able to reach ES nodes , check below

[root@iacapps-logstash-node1 ~]# curl --cacert /etc/logstash/certs/ca.crt  -u logstash_system  https://10.62.4.15:9200
Enter host password for user 'logstash_system':
{
  "name" : "iacapps-es-node1",
  "cluster_name" : "iacapps-elastic",
  "cluster_uuid" : "KyJpUNUxR3a0Fl5HOF0-qQ",
  "version" : {
    "number" : "6.2.3",
    "build_hash" : "c59ff00",
    "build_date" : "2018-03-13T10:06:29.741383Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
[root@iacapps-logstash-node4 ~]#  curl --cacert /etc/logstash/certs/ca.crt  -u logstash_system  https://10.62.4.15:9200
Enter host password for user 'logstash_system':
{
  "name" : "iacapps-es-node1",
  "cluster_name" : "iacapps-elastic",
  "cluster_uuid" : "KyJpUNUxR3a0Fl5HOF0-qQ",
  "version" : {
    "number" : "6.2.3",
    "build_hash" : "c59ff00",
    "build_date" : "2018-03-13T10:06:29.741383Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
13

Reset the password , its working now
Not sure if that was the issue

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.