Trying to bring up filebeat + logstash + Elasticsea + Kibana

Hi!
Followed Elasticsearch docs while installing elasticsearch + kibana + logstash + filebeat default set up.
For the moment filebeat configured to send events using logstash-tutorial.log.gz (extracted to logstash-tutorial.log )

filebeat.yml config file

filebeat.inputs:
- type: log
  paths:
    - /root/logstash-tutorial.log
output.logstash:
  hosts: ["192.168.0.36:5044"]
[root@fbeat filebeat]# filebeat test config

Config OK

root@fbeat filebeat]# filebeat test output
logstash: 192.168.0.36:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.0.36
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

logstash config file

initially I put simple first-pipeline.conf file in /etc/logstash/conf.d

input {
  beats {
    port => 5044
  }
}

output {
  stdout { codec => rubydebug }
}

run

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/first-pipeline.conf –config.test_and_exit

got output - configuration - OK, and then run

/usr/share/logstash/bin/logstash
-f /etc/logstash/conf.d/first-pipeline.conf --config.reload.automatic`

and was able to see some event lines as described in ELK docs

also run

./logstash -e 'input {stdin{}} output {stdout{}}'    

and got Hello world output successfully. I assume logstash works normally after installtion

elasticsearch and kibana were installed

elasticsearch config

cluster.name: mycluster
node.name: testnode
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["logstash"]
http.host: 0.0.0.0

kibana config file

logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file


pid.file: /run/kibana/kibana.pid

elasticsearch.hosts: ['https://192.168.0.36:9200']
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE2NzY5MzM5NjQyNTU6SV9mTXhBcDdRWWEycWxGeDJTcEdpQQ
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1676933965095.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://192.168.0.36:9200'], ca_trusted_fingerprint: e6a2f4089aca4420b66a653bb1fbbfae7c1e57546d7ce30f98aa13c5e6c8f9a4}]

opened browser, input token, elastic password and I can see kibana web interface
Also I can go to browser and https://localhost:9200 and can see output from elasticsearch. Kibana and elasticsearch are able to communicate and look good to me.

Now changed in logstash folder /conf.d I removed first-pipeline.conf, copied logstash-sample.conf and have next configuration

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "Mi0Kg1Wsd_zM=RIkF2f6"
  }
}

in logstash-palin.log I can see lines

[2023-02-21T11:02:10,679][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2023-02-21T11:02:15,729][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"localhost:9200 failed to respond", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::OrgApacheHttp::NoHttpResponseException: localhost:9200 failed to respond>}
[2023-02-21T11:02:15,730][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}

logstash is not able to communicate to elasticsearch though I provided username and password. Before, when I did the same with sending events directly from filebeat to elasticsearch it was working just fine
I tried to change

  hosts => ["http://localhost:9200"]

to

  hosts => ["https://localhost:9200"]

but logstash failed to load, obviously it doesn't accept the changed line

What should I change in logstash config to communicate to elasticsearch?
How can I check if elasticsearch receives events from logstash? in elasticsearch log file?

Thank you

Your Elasticsearch is configured to use https, so you need to use https in logstash configuration, what do you have in Logstash logs after you set it to use https ? You need to share the logs.

Hi!
I've restarted logstash and now it was able to run but still can see error message:
from #systemctl status logstash -l | grep -i error

Feb 21 14:15:24 logstash logstash[3271]: [2023-02-21T14:15:24,069][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://localhost:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
Feb 21 14:15:29 logstash logstash[3271]: [2023-02-21T14:15:29,144][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://localhost:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

from logstash-plain.log

[2023-02-21T14:18:01,790][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
[2023-02-21T14:18:01,795][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://localhost:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

You need to configure the certificate authority you used to create your certificates in the Elasticsearch output.

Check this documentation.

Basically you will need to add cacert => "/parth/to/the/ca.crt" to the elasticsearch output.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.