Filebeat is not sending data to logstash


(Aditya Patawari) #1

Hello,

I am using filebeat to send logs to logstash. The logs get written all the time from the application but filebeat is not able to detect it.

Here is the filebeat version

# filebeat --version
filebeat version 1.0.1 (amd64)

Here are the log lines from filebeat debug:

Oct 14 07:20:08 python-app /usr/bin/filebeat[11635]: prospector.go:186: Start next scan
Oct 14 07:20:08 python-app /usr/bin/filebeat[11635]: prospector.go:207: scan path /home/deployer/datascience-djangoservices/datascience/logs/*.log
Oct 14 07:20:08 python-app /usr/bin/filebeat[11635]: prospector.go:219: Check file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 14 07:20:08 python-app /usr/bin/filebeat[11635]: prospector.go:341: Update existing file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 14 07:20:08 python-app /usr/bin/filebeat[11635]: prospector.go:383: Not harvesting, file didn't change: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log

The timestamp of the logfile is getting updated:

# ls -l /home/deployer/datascience-djangoservices/datascience/logs/*.log
-rw-rw-rw- 1 deployer devops 247461867 Oct 14 07:21 /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log

Can anyone point me in the right direction for debugging?


(Tudor Golubenco) #2

1.0.1 is quite old now. Can you upgrade to a more recent version, please? You can go for 1.3.1 or even 5.0.0-rc1 which will soon become the stable version.


(Aditya Patawari) #3

I have tried with version 1.3.1 for another server hosting an instance of the same app. I am getting the same issue:

Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:185: Start next scan
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:261: scan path /var/log/uwsgi/*.log
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:261: scan path /home/deployer/datascience-djangoservices/datascience/logs/*.log
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:275: Check file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: registrar.go:175: Same file as before found. Fetch the state.
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:418: Update existing file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 14 10:23:22 python-app2 /usr/bin/filebeat[30070]: prospector.go:465: Not harvesting, file didn't change: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log

(Tudor Golubenco) #4

For this 1.3.1 instance, can you post:

  • your configuration file
  • the contents of the registry file (normally it's found under /var/lib/filebeat/registry)
  • a stat on the /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log file, taken at approximately the same time as when you snapshot the registry file.

(Aditya Patawari) #5

Here is the config:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/uwsgi/*.log 
        - /home/deployer/datascience-djangoservices/datascience/logs/*.log
      input_type: log
  registry_file: /var/lib/filebeat/registry
output:
  logstash:
    hosts: ["<hostname>:5548"]
shipper:
logging:
  files:
  level: debug

here is the registry file:

$ cat /var/lib/filebeat/registry
{"/home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log":{"source":"/home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log","offset":0,"FileStateOS":{"inode":274557,"device":51713}},"/var/log/uwsgi/datascience.log":{"source":"/var/log/uwsgi/datascience.log","offset":0,"FileStateOS":{"inode":272293,"device":51713}}}

Here is the output of stat command:

$ stat /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
  File: ‘/home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log’
  Size: 346550218 	Blocks: 676864     IO Block: 4096   regular file
Device: ca01h/51713d	Inode: 274557      Links: 1
Access: (0666/-rw-rw-rw-)  Uid: ( 1001/deployer)   Gid: ( 1002/  devops)
Access: 2016-10-14 11:30:28.397692800 +0000
Modify: 2016-10-14 11:30:28.397692800 +0000
Change: 2016-10-14 11:30:28.397692800 +0000
 Birth: -

(Aditya Patawari) #6

Hello,

Can anyone help me out here?


(Tudor Golubenco) #7

Sorry for the delay. Two things:

  • Please check that filebeat has write rights to the registry file
  • Please post again the Filebeat log at debug level. 1.3.1 prints more messages, so perhaps those can hint at the problem

(Aditya Patawari) #8

Filebeat has full permissions. Here is the log from the log file:

Oct 24 08:38:07 python-app2 /usr/bin/filebeat[15737]: client.go:124: 0 events out of 2048 events sent to logstash. Continue sending ...
Oct 24 08:38:07 python-app2 /usr/bin/filebeat[15737]: client.go:146: Try to publish 2048 events to logstash with window size 1
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:185: Start next scan
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:261: scan path /var/log/uwsgi/*.log
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:261: scan path /home/deployer/datascience-djangoservices/datascience/logs/*.log
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:275: Check file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: registrar.go:175: Same file as before found. Fetch the state.
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:418: Update existing file for harvesting: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log
Oct 24 08:38:13 python-app2 /usr/bin/filebeat[15737]: prospector.go:465: Not harvesting, file didn't change: /home/deployer/datascience-djangoservices/datascience/logs/datascience_apis.log

(ruflin) #9

It seems like there is an issue with the connection to Logstash as in the above log it states 0 out of 1024 messages sent to logstash. Can you share some more details on your LS setup?


(Aditya Patawari) #10

Here is the logtash config:

input {
  beats {
    port => 5544
    tags => ["rr"]
  }
  beats {
    port => 5545
    tags => ["rr"]
  }
  beats {
    port => 5546
    tags => ["to"]
  }
  beats {
    port => 5548
    tags => ["ds"]
  }
}


output {
  if "rr" in [tags] {
    elasticsearch { hosts => ["10.0.4.125:9200"] }
    file {
      path => "/ebs/logs/%{host}/%{+YYYY-MM-dd}/%{source}"
    }
  } else if "to" in [tags] {
    elasticsearch { 
      hosts => ["10.0.4.125:9200"]
      index => "to-logstash-%{+YYYY.MM.dd}"
    }
  } else if "ds" in [tags] {
    elasticsearch {
      hosts => ["10.0.4.125:9200"]
      index => "ds-logstash-%{+YYYY.MM.dd}"
    }
  }
}

From the application server, I can successfully telnet to the port 5548.


(ruflin) #11

Do you see any errors or more information in the logstash log? Is there some more information in the filebeat log messages especially on startup on why the connection to LS does not work?


(system) #12

This topic was automatically closed after 21 days. New replies are no longer allowed.