Logstash don't send logs to elasticsearch

Elastic Stack Logstash
1

Hi community,
I configured ssl on the whole stack (elasticsearch, kibana, logstash, filebeat)
When I manually launch logstash to test my configuration, I have this return.
I think it's good but I don't see anything in the discover part of kibana...
Could you help me?

Best regards,

sudo -u root /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t > /opt/error-logstash.log

Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2021-07-10T19:02:26,373][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2021-07-10T19:02:26,551][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[2021-07-10T19:02:30,462][INFO ][org.reflections.Reflections] Reflections took 243 ms to scan 1 urls, producing 24 keys and 48 values
Configuration OK
[2021-07-10T19:02:31,615][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

2

You need to provide more information.

What is not working? What do you have in the logs?

How are you starting logstash? Are you running it as a service?

And what is your configuration that is not working?

3

That tells logstash to verify the configuration is valid and then exit without processing any data. That is exactly what it is doing

Remove the -t

4

@leandrojmp
What is not working ?
I don't see the logs in kibana.

What do you have in the logs ?
It's sample of logs just for try, I followed this tutorial: How to setup TLS for Elasticsearch, Kibana, Logstash and Filebeat with offline install in Linux | Elastic Blog

How are you starting logstash? Are you running it as a service?
I start logstash as a service yes

And what is your configuration that is not working?
All my services run, just in logstash/logstash-plain.log I don't have the same return as when I run it manually. The logs don't tell me that the configuration is ok.
And when i want to check the ssl with a curl i have this return

curl -k --cacert /etc/logstash/config/certs/ca.crt https://node1.ad-it.fr:5044
curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Thanks

5

@Badger I removed the -t but I still don't see any logs in kibana.
Thanks

6

If you are following the tutorial and running logstash as a service then when you run it on the command line (without -t) I would expect an error telling you port 5044 is already in use. If you are not getting that then the service is probably not running.

What does your logstash configuration file look like?

7

@Badger Thanks for your reply.
Indeed i have the error "adress already in use" in /opt/error-logstash.log

This is the uncommented lines in logstash.yml

node.name: logstash
path.data: /var/lib/logstash
pipeline.ordered: auto
path.config: "/etc/logstash/conf.d/example.conf"
http.host: 192.168.1.30
http.port: 9600-9700
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: 'logstash_system'
xpack.monitoring.elasticsearch.password: 'LoGsTasH903'
xpack.monitoring.elasticsearch.hosts: ["https://node1.ad-it.fr:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/config/certs/ca.crt

conf.d/example.conf

input {
beats {
type => beats
host => "node1.ad-it.fr"
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/logstash/config/certs/ca.crt"]
ssl_key => "/etc/logstash/config/certs/logstash.pkcs8.key"
ssl_certificate => "/etc/logstash/config/certs/logstash.crt"
ssl_verify_mode => "force_peer"

}
}
output {
elasticsearch {
hosts => ["https://node1.ad-it.fr:9200"]
ssl => true
cacert => '/etc/logstash/config/certs/ca.crt'
user => 'logstash_writer'
password => 'changeme'
index => 'logstash-%{+YYYY.MM.dd}'
}
}

8

What beats have you configured? There is not going to be any data in kibana unless a beat sends some.

9

@Badger Thanks
I have configured a filebeat on the same machine with kibana and logstash.
The filebeat service is running, it says "started Filebeat sends log files to Logstash or directly to Elasticsearch.."

This is my uncommented lines in filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /etc/filebeat/logstash-tutorial-dataset
#============================= Filebeat modules ===============================
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 3
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["node1-ad-it.fr:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/filebeat/config/certs/ca.crt"]

  # Certificate for SSL client authentication
  ssl.certificate: "/etc/logstash/config/certs/logstash.crt"

  # Client Certificate Key
  ssl.key: "/etc/logstash/config/certs/logstash.pkcs8.key"
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
10

The indentation looks wrong there. There should be two spaces before enabled and paths.

Is logstash-tutorial-dataset a file or a directory?

11

@Badger
I'm sorry there is actually 2 spaces in my config file.
logstash-tutorial-dataset is a directory.
Thanks

12

Can you try /etc/filebeat/logstash-tutorial-dataset/*?

13

@Badger I put /etc/filebeat/logstash-tutorial-dataset/* in filebeat.yml it is still not working

When in curl logstash, i have this error, you think it is an issue? (self-signed certificate with certutil)
curl -k --cacert /etc/logstash/config/certs/ca.crt https://node1.ad-it.fr:5044
curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Thanks

14

I believe this is logstash is telling filebeat that it does not trust the client certificate that filebeat sent. I suggest you comment out

ssl_verify_mode => "force_peer"

in the logstash configuration and

ssl.certificate: "/etc/logstash/config/certs/logstash.crt"

in the filebeat configuraton. If that works you have confirmed the client certificate is the problem. I would then start checking that you have certificate in the right formats, that the private keys match, etc.

15

@Badger
When i comment this line "ssl.certificate: "/etc/logstash/config/certs/logstash.crt" logstash service remains stuck in restart.
Thanks

16

Commenting out that line in the filebeat configuration should not affect logstash at all.

17

@Badger
My bad i restarted my machine and then i was able to restart the services with thoses lines commented. I still have the same problem (no logs in kibana and bad certificate with curl).
Thanks

18

If it is failing to connect then the filebeat logs should indicate why. It will only connect when it has new data to send.

19

@badger thanks
In filebeat logs it is written :
What should I understand?

|2021-07-11T02:03:27.499+0200|INFO|log/harvester.go:255|Harvester started for file: /etc/filebeat/logstash-tutorial-dataset/logstash-tutorial.log|
|---|---|---|---|
|2021-07-11T02:03:27.501+0200|INFO|crawler/crawler.go:106|Loading and starting Inputs completed. Enabled inputs: 1|
|2021-07-11T02:03:27.501+0200|INFO|cfgfile/reload.go:150|Config reloader started|
|2021-07-11T02:03:27.501+0200|INFO|cfgfile/reload.go:205|Loading of config files completed.|
|2021-07-11T02:03:30.490+0200|INFO|add_cloud_metadata/add_cloud_metadata.go:340|add_cloud_metadata: hosting provider type not detected.|
|2021-07-11T02:03:31.490+0200|INFO|pipeline/output.go:95|Connecting to backoff(async(tcp://logstash.ad-it.fr:5044))
2021-07-11T02:03:33.237+0200	ERROR	pipeline/output.go:100	Failed to connect to backoff(async(tcp://logstash.ad-it.fr:5044)): x509: certificate signed by unknown authority
2021-07-11T02:03:33.237+0200	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(async(tcp://logstash.ad-it.fr:5044)) with 1 reconnect attempt(s)
|
20

I think that is saying that

ssl.certificate_authorities: ["/etc/filebeat/config/certs/ca.crt"]

does not include the certificate used to sign the certificate that logstash uses.