Logstash don't send logs to elasticsearch

diegz · July 10, 2021, 5:06pm

Hi community,
I configured ssl on the whole stack (elasticsearch, kibana, logstash, filebeat)
When I manually launch logstash to test my configuration, I have this return.
I think it's good but I don't see anything in the discover part of kibana...
Could you help me?

Best regards,

sudo -u root /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t > /opt/error-logstash.log

Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2021-07-10T19:02:26,373][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2021-07-10T19:02:26,551][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[2021-07-10T19:02:30,462][INFO ][org.reflections.Reflections] Reflections took 243 ms to scan 1 urls, producing 24 keys and 48 values
Configuration OK
[2021-07-10T19:02:31,615][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

leandrojmp · July 10, 2021, 5:13pm

You need to provide more information.

What is not working? What do you have in the logs?

How are you starting logstash? Are you running it as a service?

And what is your configuration that is not working?

Badger · July 10, 2021, 5:14pm

That tells logstash to verify the configuration is valid and then exit without processing any data. That is exactly what it is doing

Remove the -t

diegz · July 10, 2021, 7:28pm

@leandrojmp
What is not working ?
I don't see the logs in kibana.

What do you have in the logs ?
It's sample of logs just for try, I followed this tutorial: How to setup TLS for Elasticsearch, Kibana, Logstash and Filebeat with offline install in Linux | Elastic Blog

How are you starting logstash? Are you running it as a service?
I start logstash as a service yes

And what is your configuration that is not working?
All my services run, just in logstash/logstash-plain.log I don't have the same return as when I run it manually. The logs don't tell me that the configuration is ok.
And when i want to check the ssl with a curl i have this return

curl -k --cacert /etc/logstash/config/certs/ca.crt https://node1.ad-it.fr:5044
curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Thanks

diegz · July 10, 2021, 7:30pm

@Badger I removed the -t but I still don't see any logs in kibana.
Thanks

Badger · July 10, 2021, 7:51pm

If you are following the tutorial and running logstash as a service then when you run it on the command line (without -t) I would expect an error telling you port 5044 is already in use. If you are not getting that then the service is probably not running.

What does your logstash configuration file look like?

diegz · July 10, 2021, 8:09pm

@Badger Thanks for your reply.
Indeed i have the error "adress already in use" in /opt/error-logstash.log

This is the uncommented lines in logstash.yml

node.name: logstash
path.data: /var/lib/logstash
pipeline.ordered: auto
path.config: "/etc/logstash/conf.d/example.conf"
http.host: 192.168.1.30
http.port: 9600-9700
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: 'logstash_system'
xpack.monitoring.elasticsearch.password: 'LoGsTasH903'
xpack.monitoring.elasticsearch.hosts: ["https://node1.ad-it.fr:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/config/certs/ca.crt

conf.d/example.conf

input {
beats {
type => beats
host => "node1.ad-it.fr"
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/logstash/config/certs/ca.crt"]
ssl_key => "/etc/logstash/config/certs/logstash.pkcs8.key"
ssl_certificate => "/etc/logstash/config/certs/logstash.crt"
ssl_verify_mode => "force_peer"

}
}
output {
elasticsearch {
hosts => ["https://node1.ad-it.fr:9200"]
ssl => true
cacert => '/etc/logstash/config/certs/ca.crt'
user => 'logstash_writer'
password => 'changeme'
index => 'logstash-%{+YYYY.MM.dd}'
}
}

Badger · July 10, 2021, 8:23pm

What beats have you configured? There is not going to be any data in kibana unless a beat sends some.

diegz · July 10, 2021, 8:35pm

@Badger Thanks
I have configured a filebeat on the same machine with kibana and logstash.
The filebeat service is running, it says "started Filebeat sends log files to Logstash or directly to Elasticsearch.."

This is my uncommented lines in filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /etc/filebeat/logstash-tutorial-dataset
#============================= Filebeat modules ===============================
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 3
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["node1-ad-it.fr:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/filebeat/config/certs/ca.crt"]

  # Certificate for SSL client authentication
  ssl.certificate: "/etc/logstash/config/certs/logstash.crt"

  # Client Certificate Key
  ssl.key: "/etc/logstash/config/certs/logstash.pkcs8.key"
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Badger · July 10, 2021, 8:42pm

The indentation looks wrong there. There should be two spaces before enabled and paths.

Is logstash-tutorial-dataset a file or a directory?

diegz · July 10, 2021, 8:48pm

@Badger
I'm sorry there is actually 2 spaces in my config file.
logstash-tutorial-dataset is a directory.
Thanks

Badger · July 10, 2021, 9:21pm

Can you try /etc/filebeat/logstash-tutorial-dataset/*?

diegz · July 10, 2021, 9:40pm

@Badger I put /etc/filebeat/logstash-tutorial-dataset/* in filebeat.yml it is still not working

When in curl logstash, i have this error, you think it is an issue? (self-signed certificate with certutil)
curl -k --cacert /etc/logstash/config/certs/ca.crt https://node1.ad-it.fr:5044
curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Thanks

Badger · July 10, 2021, 9:51pm

I believe this is logstash is telling filebeat that it does not trust the client certificate that filebeat sent. I suggest you comment out

ssl_verify_mode => "force_peer"

in the logstash configuration and

ssl.certificate: "/etc/logstash/config/certs/logstash.crt"

in the filebeat configuraton. If that works you have confirmed the client certificate is the problem. I would then start checking that you have certificate in the right formats, that the private keys match, etc.

diegz · July 10, 2021, 10:31pm

@Badger
When i comment this line "ssl.certificate: "/etc/logstash/config/certs/logstash.crt" logstash service remains stuck in restart.
Thanks

Badger · July 10, 2021, 10:34pm

Commenting out that line in the filebeat configuration should not affect logstash at all.

diegz · July 10, 2021, 11:00pm

@Badger
My bad i restarted my machine and then i was able to restart the services with thoses lines commented. I still have the same problem (no logs in kibana and bad certificate with curl).
Thanks

Badger · July 10, 2021, 11:47pm

If it is failing to connect then the filebeat logs should indicate why. It will only connect when it has new data to send.

diegz · July 11, 2021, 12:06am

@badger thanks
In filebeat logs it is written :
What should I understand?

|2021-07-11T02:03:27.499+0200|INFO|log/harvester.go:255|Harvester started for file: /etc/filebeat/logstash-tutorial-dataset/logstash-tutorial.log|
|---|---|---|---|
|2021-07-11T02:03:27.501+0200|INFO|crawler/crawler.go:106|Loading and starting Inputs completed. Enabled inputs: 1|
|2021-07-11T02:03:27.501+0200|INFO|cfgfile/reload.go:150|Config reloader started|
|2021-07-11T02:03:27.501+0200|INFO|cfgfile/reload.go:205|Loading of config files completed.|
|2021-07-11T02:03:30.490+0200|INFO|add_cloud_metadata/add_cloud_metadata.go:340|add_cloud_metadata: hosting provider type not detected.|
|2021-07-11T02:03:31.490+0200|INFO|pipeline/output.go:95|Connecting to backoff(async(tcp://logstash.ad-it.fr:5044))
2021-07-11T02:03:33.237+0200	ERROR	pipeline/output.go:100	Failed to connect to backoff(async(tcp://logstash.ad-it.fr:5044)): x509: certificate signed by unknown authority
2021-07-11T02:03:33.237+0200	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(async(tcp://logstash.ad-it.fr:5044)) with 1 reconnect attempt(s)
|

Badger · July 11, 2021, 12:25am

I think that is saying that

ssl.certificate_authorities: ["/etc/filebeat/config/certs/ca.crt"]

does not include the certificate used to sign the certificate that logstash uses.

Topic		Replies	Views
No logstash pipeline Logstash elastic-stack-security	5	1338	July 21, 2021
Logstash "Received fatal alert : bad_certificate" Logstash elastic-stack-security	2	2795	July 23, 2021
Logs not pushed to elasticsearch Logstash	6	629	May 30, 2023
Logstash config advise Logstash	22	55	October 9, 2024
Logstash configurations for v8.12 Logstash	2	120	March 12, 2024

Logstash don't send logs to elasticsearch

Related Topics