Filebeat doesn't send data to logstash (?) OR logstash isn't receiving it (?)

[boxclever38 (Chris)]

I have an Elastic stack working A-OK (afaik). Filebeat runs great on the local machine, sending to logstash, and everything looks good in Kibana.

I am now trying to send logs from a remote machine and it does not appear to work, but I can't find any errors. This makes me think I have something misconfigured.

Filebeat installation and config is identical on both servers:

filebeat.yml (on both the server and client machines)

output.logstash:
# The Logstash hosts
hosts: ["10.137.4.219:5044"]

I'm using the filebeat "system" module for both client and server machines.

Enabling filebeat on the server populates data like crazy. Enabling it on the remote client does nothing, but it shows that a connection is established:

from /var/log/filebeat/filebeat on the client machine:

|2019-03-22T11:37:32.207-0600|INFO|log/harvester.go:255|Harvester started for file: /var/log/syslog|
|---|---|---|---|
|2019-03-22T11:37:32.207-0600|INFO|log/harvester.go:255|Harvester started for file: /var/log/auth.log|
|2019-03-22T11:37:33.208-0600|INFO|pipeline/output.go:95|Connecting to backoff(async(tcp://10.137.4.219:5044))|
|2019-03-22T11:37:33.208-0600|INFO|pipeline/output.go:105|Connection to backoff(async(tcp://10.137.4.219:5044)) established|

Is this a filebeat misconfig or a logstash one? Should I be using the same config for all clients that I want to send logs over to logstash?

[pup_seba]

Hi!

Could you check if:

• Service is enabled ('systemctl status filebeat' or similar)
• "enabled" under your filebeat.yml is not commented or set to "false". It should look something like this:
  - type: log
Change to true to enable this input configuration.
enabled: true

(don't pay attention to the indentation, I'm not being able to make it look good in here :D)

[boxclever38 (Chris)]

Thanks for the reply.

#1: Yes, service is enabled on both the client and server:
Active: **active (running)**

#2: On your second point, enabled is set to false. However this is true on both machines. I'm using the system module, so it was my understanding leaving the setting as false is correct.

Here's the config on both machines:

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

I changed it to "true" on both, then restarted filebeat on both. No change detected.
logging from filebeat on the server is still A-OK
logging from filebeat on the client still non-existent

If you need any more config info, let me know.

For what it's worth, both machines both have metricbeat working without issue. The difference is I'm trying to use filebeat -> logstash -> elasticsearch but no logstash with metricbeat. It's just straight metricbeat -> elasticsearch

[pup_seba]

In my experience, that needs to be set to true.

Can you telnet your logstash and beats port? All i ever needed to make filebeat send info to logstash is to enable it in the filebeat.yml, configure the output for logstash, and being able to comunicate with logstash on the beats port configured for it.

[boxclever38 (Chris)]

Yes, I can telnet to the beats port.

I spun up a second server and was able to connect successfully. So it wasn't an issue on the Server. It has to be on the client.

This is nuts... I can't see anything wrong at all. There must be something though.

I deleted the filebeat.yml file on the problem machine, and replaced it with a known good one. Still nothing.

Is there a way to monitor the connection or see if data is going through? The log files show no errors.

[pup_seba]

If I stop logstash service, then I can see under the /var/log/filebeat/filebeat, these messages:

I'm quite confused This is how my logstash (server with ip 172.16.1.172)input looks. (/etc/logstash/conf.d/10_input.conf

input {
 beats {
   port => 5044
   ssl => true
   ssl_certificate_authorities => ["/etc/logstash/certs/logstash_ca.pem"]
   ssl_certificate => "/etc/logstash/certs/elasticstack01.essi.lab.pem"
   ssl_key => "/etc/logstash/certs/elasticstack01.essi.lab.pk8"
   ssl_verify_mode => "force_peer"
 }
}

Although, it also works like this:

input {
 beats {
   port => 5044
 }
}

These are the relevant pieces of my /etc/filebeat/filebeat.yml (my client server with ip 172.16.1.68)

filebeat.inputs:
 - type: log
   enabled: true
   paths:
     - /opt/zimbra/log/mailbox.log
 multiline.pattern: ^\d{4}-\d{2}-\d{2}
 multiline.negate: true
 multiline.match: after
filebeat.config.modules:
 path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 5s
setup.template.settings:
 index.number_of_shards: 6
output.logstash:
 hosts: ["elasticstack01.essi.lab:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/certs/logstash_ca.pem"]
  ssl.certificate: "/etc/filebeat/certs/zimbrastore01.essi.lab.pem"
  ssl.key: "/etc/filebeat/certs/zimbrastore01.essi.lab.key"
processors:
 - add_host_metadata: ~
 - add_cloud_metadata: ~

Sorry the indentetation is not looking great...I still don't know how to make it look better in this forum

Again, the ssl part in this filebeat.yml, is not required unless you are using it for your logstash too. Beside that ssl thing, mine is pretty much a default installation.

Maybe you are able to find something under /var/log/filebeat/filebeat from the server where you are trying to send the logs from. Or you could try to see if you see any error in your logstash server under /var/log/logstash/logstash-plain.log.

I also see under the filebeat.yml, that you have a "debug" option, maybe you can enable it in the server you are trying to send the logs from, and see if it throws some more light, maybe?

[boxclever38 (Chris)]

Thanks for the input, appreciate it.

Turns out the problem was timezones. The client machine was set to a local timezone and the server was set to UTC. So when I was looking in Kibana and watching the dashboard no data was in the range I was specifying - it was shifted.

The odd thing is I would have expected that timezones should be accounted for, but I guess not.

On a positive note, I now know how to run filebeat and logstash in debug mode, and was able watch both in Terminal to verify that communication was happening.

For the record, the command for filebeat is:

filebeat -e -d "publish

for logstash:

usr/share/logstash/bin/logstash -f /etc/logstash/conf.d --debug

The data that tipped me off was in the output for one line of log file data. On client A:
"@timestamp" => 2019-03-26T14:51:20.000Z,
on Client B:
"@timestamp" => 2019-03-26T20:51:42.000Z,

Note the 6h difference. Make sure all clients and servers are set to the same timezone in order to avoid banging your head against a brick wall like I did.

As an aside - if you use filebeat modules you can completely comment out the "system" section of the filebeat.yml file to avoid potential conflicts.