Filebeat and Logstash no send Logs

roxasval · October 21, 2021, 9:28am

Hello to all

I'm a newbie on Elasticsearch and I'm having some problems:

I have a cluster of 2 Elasticsearch (but I do most things with one) and a separate kibana.
I would like to retrieve logs from network devices and ESXi
So I'm doing some tests with the ESXi's, sending the logs to my Elasticsearch server.
Currently I have Metricbeat installed and it works fine.
I would like to use Filebeat and/or Logstash to retrieve and send the logs to Elasticsearch
The problems are the following:

using Filebeat :
when I set Elasticsearch to output, it doesn't receive anything, but when I do filebeat test output, or filebeat test config, everything is ok, so I think it doesn't receive anything but I'm not sure and don't know how to see this.
when I set Logstash as output, I configure Logstash with Beats input, but when I do a netstat -an | grep "LISTEN", no application listens on port 5044 (I tried with different ports, but logstash never listens)

I hope that I have stated my expectations and problems, and I thank you in advance for your help

I'm attaching my current filebeat.yml

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/eol/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

filebeat.inputs:
- type: tcp
  enabled: true
  max_message_size: 10MiB
  host: "localhost:2345"
  
  
setup.kibana.host: "https://eol-kibana:5601"
setup.kibana.ssl.enabled: true
setup.kibana.ssl.certificate_authorities: ["/etc/metricbeat/certs/EoL.crt"]
setup.kibana.ssl.certificate: "/etc/metricbeat/certs/eol-elastic-01.crt"
setup.kibana.ssl.key: "/etc/metricbeat/certs/eol-elastic-01_privatekey.pem"

setup.template.enabled: false
setup.ilm.enabled: false

output.elasticsearch:
  hosts: ["https://eol-elastic-01:9200"]
  username: "elastic"
  password: "elastic"
  protocol: "https"
  ssl.certificate_authorities: ["/etc/filebeat/certs/EoL.crt"]
  index: "eol-%{[event][provider]}"
  pipeline: "eol-%{[pipeline]}-pipeline"

Thank you very much

grumo35 · October 21, 2021, 9:36am

Hello,

I'm not really experienced on elastic input and direct forwarding but can you show Fileebat logs ?

Also as of logstash please show me your configuration ?

I can show you a working listenning beats configuration :

input {
  beats {
    port => 9999
    host => "1.2.3.254"
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/ca.crt"]
    ssl_certificate => "/etc/logstash/certs/logstash1.crt"
    ssl_key => "/etc/logstash/certs/logstash1.key"
    ssl_verify_mode => "force_peer"
    codec => plain { charset=>"UTF-8" }
  }
}

roxasval · October 21, 2021, 11:38am

All right, thank you!
I tried quickly to send to logstash with your configuration but it didn't work either, the port I assign to logstash (here 5044) is not on listen when I do a netstat -an | grep "LISTEN"

Here is one of the files in the conf.d directory of my Logstash:

input {
  beats {
    port => 5044
    host => "eol-elastic-01"
  }
}

filter {
  mutate {
    copy => {
     "[fields][log_prefix]" => "[@metadata][log_prefix]"
     "[fields][log_idx]" => "[@metadata][index]"
     "[fields][application]" => "[@metadata][application]"
    }
  }
}
output {
  elasticsearch {
    user => "elastic"
    password => "elastic"
    hosts => ["eol-elastic-01:9200"]
  }
  stdout { codec => rubydebug }
}

Bien évidemment je modifie mon fichier filebeat pour envoyer vers logstash plutôt qu'Elasticsearch :

output.logstash:
  hosts: ["eol-elastic-01:5044"]

Thanks for your answer

grumo35 · October 21, 2021, 11:40am

Ok so your logstash service is started but the configuration must be wrong that's why the pipeline doesnt start.

Can you check "/var/log/logstash/logstash-plain.log" entries ?

Also a journalctl -ru logstash ( gives you the last output on top )

You can also check for systemctl status logstash.

We'll sort this out

roxasval · October 21, 2021, 12:01pm

Thanks !

 tail -f /var/log/logstash/logstash-plain.log
[2021-10-20T12:24:01,440][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/main-*.conf"}
[2021-10-20T12:24:01,462][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/dead_letter_queue-*.conf"}
[2021-10-20T12:24:01,495][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2021-10-20T12:24:02,185][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-10-20T12:24:07,146][INFO ][logstash.runner          ] Logstash shut down.
[2021-10-20T12:24:07,214][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

and for the journalctl -ru logstash :

oct. 21 13:55:59 eol-elastic-01.maquette.lan logstash[8746]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecate
oct. 21 13:55:58 eol-elastic-01.maquette.lan logstash[8746]: Using bundled JDK: /usr/share/logstash/jdk
oct. 21 13:55:58 eol-elastic-01.maquette.lan systemd[1]: Started logstash.
oct. 21 13:55:58 eol-elastic-01.maquette.lan systemd[1]: Stopped logstash.
oct. 21 13:55:58 eol-elastic-01.maquette.lan systemd[1]: logstash.service holdoff time over, scheduling restart.
oct. 21 13:55:58 eol-elastic-01.maquette.lan systemd[1]: logstash.service failed.
oct. 21 13:55:58 eol-elastic-01.maquette.lan systemd[1]: Unit logstash.service entered failed state.
oct. 21 13:55:58 eol-elastic-01.maquette.lan systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
oct. 21 13:55:58 eol-elastic-01.maquette.lan logstash[8688]: at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstas
oct. 21 13:55:58 eol-elastic-01.maquette.lan logstash[8688]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-compl
oct. 21 13:55:58 eol-elastic-01.maquette.lan logstash[8688]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-compl
oct. 21 13:55:58 eol-elastic-01.maquette.lan logstash[8688]: org.jruby.exceptions.SystemExit: (SystemExit) exit
oct. 21 13:55:58 eol-elastic-01.maquette.lan logstash[8688]: [FATAL] 2021-10-21 13:55:58.339 [main] Logstash - Logstash stopped proces
oct. 21 13:55:58 eol-elastic-01.maquette.lan logstash[8688]: ERROR: Failed to parse YAML file "/etc/logstash/logstash.yml". Please con
oct. 21 13:54:43 eol-elastic-01.maquette.lan logstash[8688]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecate
oct. 21 13:54:42 eol-elastic-01.maquette.lan logstash[8688]: Using bundled JDK: /usr/share/logstash/jdk
oct. 21 13:54:42 eol-elastic-01.maquette.lan systemd[1]: Started logstash.
oct. 21 13:54:42 eol-elastic-01.maquette.lan systemd[1]: Stopped logstash.
oct. 21 13:54:42 eol-elastic-01.maquette.lan systemd[1]: logstash.service holdoff time over, scheduling restart.
oct. 21 13:54:42 eol-elastic-01.maquette.lan systemd[1]: logstash.service failed.
oct. 21 13:54:42 eol-elastic-01.maquette.lan systemd[1]: Unit logstash.service entered failed state.
oct. 21 13:54:42 eol-elastic-01.maquette.lan systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
oct. 21 13:54:42 eol-elastic-01.maquette.lan logstash[8632]: at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstas
oct. 21 13:54:42 eol-elastic-01.maquette.lan logstash[8632]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-compl
oct. 21 13:54:42 eol-elastic-01.maquette.lan logstash[8632]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-compl
oct. 21 13:54:42 eol-elastic-01.maquette.lan logstash[8632]: org.jruby.exceptions.SystemExit: (SystemExit) exit
oct. 21 13:54:42 eol-elastic-01.maquette.lan logstash[8632]: [FATAL] 2021-10-21 13:54:42.236 [main] Logstash - Logstash stopped proces
oct. 21 13:54:42 eol-elastic-01.maquette.lan logstash[8632]: ERROR: Failed to parse YAML file "/etc/logstash/logstash.yml". Please con
oct. 21 13:53:32 eol-elastic-01.maquette.lan logstash[8632]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecate
oct. 21 13:53:32 eol-elastic-01.maquette.lan logstash[8632]: Using bundled JDK: /usr/share/logstash/jdk
oct. 21 13:53:32 eol-elastic-01.maquette.lan systemd[1]: Started logstash.
oct. 21 13:53:32 eol-elastic-01.maquette.lan systemd[1]: Stopped logstash.
oct. 21 13:53:32 eol-elastic-01.maquette.lan systemd[1]: logstash.service holdoff time over, scheduling restart.
oct. 21 13:53:32 eol-elastic-01.maquette.lan systemd[1]: logstash.service failed.
oct. 21 13:53:32 eol-elastic-01.maquette.lan systemd[1]: Unit logstash.service entered failed state.
oct. 21 13:53:32 eol-elastic-01.maquette.lan systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE

It seems that I have a problem with logstash, and I hadn't seen it, as it is "active" when I do a systemctl logstash status

I hope so! I've been working on it for a few days and I'm doing a lot of testing, I was starting to despair

grumo35 · October 22, 2021, 7:44am

Here you are, you need to take care of this error first please check for tabs or missing key values in your YAML syntax.

The logstash service will try to restart indefinitly even if the pipelines are in error, some might start.

That's how the "logstash service" works you can also launch logstash by command line to "feel" it directly if it's working.

Some times you'll have to kill the main PID in order to restart it properly it can get stuck.

roxasval · October 25, 2021, 7:28am

Hi !
Thanks ! I've check and I've found the problem with the syntax error but I've always an error, this time with the Dead_queue_letter if I remember correctly.
But, on an other hand, Filebeat works ! Filebeat can send logs to Elasticsearch.
Apparently, by commenting on these 2 lines on the section output.Elasticsearch to my Filebeat.yml file, it works:

  index: "eol-%{[event][provider]}"
  pipeline: "eol-%{[pipeline]}-pipeline"

So my problem with Filebeat came from there

I'm going to be a bit busy soon but as soon as I have time I'll have a closer look at Logstash
I'll keep you posted, and thanks again!

system · November 22, 2021, 9:28am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logs not being sent to filebeats and logstash Beats filebeat	3	256	August 13, 2023
Filebeat not sending logs to E.L.K server’s logstash Beats	5	1144	October 19, 2016
Can't send logs from filebeats to logstash Beats filebeat	5	4204	September 21, 2019
Filebeat not sending data to Elasticsearch or Logstash Beats filebeat	3	7413	May 2, 2018
Cannot see Filebeats - logstash data in Kibana Beats filebeat	5	2300	July 11, 2017

Filebeat and Logstash no send Logs

Related topics