Hi!
Followed Elasticsearch docs while installing elasticsearch + kibana + logstash + filebeat default set up.
For the moment filebeat configured to send events using logstash-tutorial.log.gz (extracted to logstash-tutorial.log )
filebeat.yml config file
filebeat.inputs:
- type: log
paths:
- /root/logstash-tutorial.log
output.logstash:
hosts: ["192.168.0.36:5044"]
[root@fbeat filebeat]# filebeat test config
Config OK
root@fbeat filebeat]# filebeat test output
logstash: 192.168.0.36:5044...
connection...
parse host... OK
dns lookup... OK
addresses: 192.168.0.36
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
logstash config file
initially I put simple first-pipeline.conf file in /etc/logstash/conf.d
input {
beats {
port => 5044
}
}
output {
stdout { codec => rubydebug }
}
run
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/first-pipeline.conf –config.test_and_exit
got output - configuration - OK, and then run
/usr/share/logstash/bin/logstash
-f /etc/logstash/conf.d/first-pipeline.conf --config.reload.automatic`
and was able to see some event lines as described in ELK docs
also run
./logstash -e 'input {stdin{}} output {stdout{}}'
and got Hello world output successfully. I assume logstash works normally after installtion
elasticsearch and kibana were installed
elasticsearch config
cluster.name: mycluster
node.name: testnode
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["logstash"]
http.host: 0.0.0.0
kibana config file
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
pid.file: /run/kibana/kibana.pid
elasticsearch.hosts: ['https://192.168.0.36:9200']
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE2NzY5MzM5NjQyNTU6SV9mTXhBcDdRWWEycWxGeDJTcEdpQQ
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1676933965095.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://192.168.0.36:9200'], ca_trusted_fingerprint: e6a2f4089aca4420b66a653bb1fbbfae7c1e57546d7ce30f98aa13c5e6c8f9a4}]
opened browser, input token, elastic password and I can see kibana web interface
Also I can go to browser and https://localhost:9200 and can see output from elasticsearch. Kibana and elasticsearch are able to communicate and look good to me.
Now changed in logstash folder /conf.d I removed first-pipeline.conf, copied logstash-sample.conf and have next configuration
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "elastic"
password => "Mi0Kg1Wsd_zM=RIkF2f6"
}
}
in logstash-palin.log I can see lines
[2023-02-21T11:02:10,679][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
[2023-02-21T11:02:15,729][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"localhost:9200 failed to respond", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::OrgApacheHttp::NoHttpResponseException: localhost:9200 failed to respond>}
[2023-02-21T11:02:15,730][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}
logstash is not able to communicate to elasticsearch though I provided username and password. Before, when I did the same with sending events directly from filebeat to elasticsearch it was working just fine
I tried to change
hosts => ["http://localhost:9200"]
to
hosts => ["https://localhost:9200"]
but logstash failed to load, obviously it doesn't accept the changed line
What should I change in logstash config to communicate to elasticsearch?
How can I check if elasticsearch receives events from logstash? in elasticsearch log file?
Thank you