Filebeat is not sending data to logstash

adimania · October 14, 2016, 7:24am

Hello,

I am using filebeat to send logs to logstash. The logs get written all the time from the application but filebeat is not able to detect it.

Here is the filebeat version

# filebeat --version
filebeat version 1.0.1 (amd64)

Here are the log lines from filebeat debug:

Oct 14 07:20:08 python-app /usr/bin/filebeat[11635]: prospector.go:186: Start next scan
Oct 14 07:20:08 python-app /usr/bin/filebeat[11635]: prospector.go:207: scan path /home/deployer/datascience-djangoservices/datascience/logs/*.log
Oct 14 07:20:08 python-app /usr/bin/filebeat[11635]: prospector.go:219: Check file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 14 07:20:08 python-app /usr/bin/filebeat[11635]: prospector.go:341: Update existing file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 14 07:20:08 python-app /usr/bin/filebeat[11635]: prospector.go:383: Not harvesting, file didn't change: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log

The timestamp of the logfile is getting updated:

# ls -l /home/deployer/datascience-djangoservices/datascience/logs/*.log
-rw-rw-rw- 1 deployer devops 247461867 Oct 14 07:21 /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log

Can anyone point me in the right direction for debugging?

tudor · October 14, 2016, 10:04am

1.0.1 is quite old now. Can you upgrade to a more recent version, please? You can go for 1.3.1 or even 5.0.0-rc1 which will soon become the stable version.

adimania · October 14, 2016, 10:26am

I have tried with version 1.3.1 for another server hosting an instance of the same app. I am getting the same issue:

Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:185: Start next scan
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:261: scan path /var/log/uwsgi/*.log
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:261: scan path /home/deployer/datascience-djangoservices/datascience/logs/*.log
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:275: Check file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: registrar.go:175: Same file as before found. Fetch the state.
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:418: Update existing file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:465: Not harvesting, file didn't change: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log

tudor · October 14, 2016, 10:41am

For this 1.3.1 instance, can you post:

your configuration file
the contents of the registry file (normally it's found under /var/lib/filebeat/registry)
a stat on the /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log file, taken at approximately the same time as when you snapshot the registry file.

adimania · October 14, 2016, 11:38am

Here is the config:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/uwsgi/*.log 
        - /home/deployer/datascience-djangoservices/datascience/logs/*.log
      input_type: log
  registry_file: /var/lib/filebeat/registry
output:
  logstash:
    hosts: ["<hostname>:5548"]
shipper:
logging:
  files:
  level: debug

here is the registry file:

$ cat /var/lib/filebeat/registry
{"/home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log":{"source":"/home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log","offset":0,"FileStateOS":{"inode":274557,"device":51713}},"/var/log/uwsgi/datascience.log":{"source":"/var/log/uwsgi/datascience.log","offset":0,"FileStateOS":{"inode":272293,"device":51713}}}

Here is the output of stat command:

$ stat /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
  File: ‘/home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log’
  Size: 346550218 	Blocks: 676864     IO Block: 4096   regular file
Device: ca01h/51713d	Inode: 274557      Links: 1
Access: (0666/-rw-rw-rw-)  Uid: ( 1001/deployer)   Gid: ( 1002/  devops)
Access: 2016-10-14 11:30:28.397692800 +0000
Modify: 2016-10-14 11:30:28.397692800 +0000
Change: 2016-10-14 11:30:28.397692800 +0000
 Birth: -

adimania · October 24, 2016, 6:28am

Hello,

Can anyone help me out here?

tudor · October 24, 2016, 8:30am

Sorry for the delay. Two things:

Please check that filebeat has write rights to the registry file
Please post again the Filebeat log at debug level. 1.3.1 prints more messages, so perhaps those can hint at the problem

adimania · October 24, 2016, 8:41am

Filebeat has full permissions. Here is the log from the log file:

Oct 24 08:38:07 python-app2 /usr/bin/filebeat[15737]: client.go:124: 0 events out of 2048 events sent to logstash. Continue sending ...
Oct 24 08:38:07 python-app2 /usr/bin/filebeat[15737]: client.go:146: Try to publish 2048 events to logstash with window size 1
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:185: Start next scan
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:261: scan path /var/log/uwsgi/*.log
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:261: scan path /home/deployer/datascience-djangoservices/datascience/logs/*.log
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:275: Check file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: registrar.go:175: Same file as before found. Fetch the state.
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:418: Update existing file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:465: Not harvesting, file didn't change: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log

ruflin · October 24, 2016, 9:33am

It seems like there is an issue with the connection to Logstash as in the above log it states 0 out of 1024 messages sent to logstash. Can you share some more details on your LS setup?

adimania · October 24, 2016, 9:52am

Here is the logtash config:

input {
  beats {
    port => 5544
    tags => ["rr"]
  }
  beats {
    port => 5545
    tags => ["rr"]
  }
  beats {
    port => 5546
    tags => ["to"]
  }
  beats {
    port => 5548
    tags => ["ds"]
  }
}


output {
  if "rr" in [tags] {
    elasticsearch { hosts => ["10.0.4.125:9200"] }
    file {
      path => "/ebs/logs/%{host}/%{+YYYY-MM-dd}/%{source}"
    }
  } else if "to" in [tags] {
    elasticsearch { 
      hosts => ["10.0.4.125:9200"]
      index => "to-logstash-%{+YYYY.MM.dd}"
    }
  } else if "ds" in [tags] {
    elasticsearch {
      hosts => ["10.0.4.125:9200"]
      index => "ds-logstash-%{+YYYY.MM.dd}"
    }
  }
}

From the application server, I can successfully telnet to the port 5548.

ruflin · October 25, 2016, 7:16am

Do you see any errors or more information in the logstash log? Is there some more information in the filebeat log messages especially on startup on why the connection to LS does not work?

system · November 4, 2016, 7:24am

This topic was automatically closed after 21 days. New replies are no longer allowed.

Topic		Replies	Views
Filebeat is not sending log entries/logs to logstash server Beats filebeat	2	928	November 30, 2018
Filebeat harvests files but doesn't send them anywhere Beats filebeat	2	573	April 16, 2019
Filebeat is not sending data to logstash from windows client Logstash	2	1197	January 10, 2018
Filebeat is not sending the logs to the logstash Beats filebeat	6	1526	August 22, 2018
Standard "Filebeat won't send to Logstash" question Beats filebeat	5	364	May 9, 2018

Filebeat is not sending data to logstash

Related topics