Filebeat won't read log files

Elastic Stack Beats
1

My filebeat doesn't read log files to send to logstash on a remote server.
Here is my config file:
filebeat.inputs:

  • type: log
    enabled: true
    paths:
    -/var/log/demisto/*.log
    logging.level: debug
    logging.to_files: true
    logging.files:
    path: /home/cloud-user/filebeat/filebeat.log
    name: filebeat
    keepfiles: 10
    permissions: 0644
    output.console:
    pretty:true
    output.logstash:
    hosts: ["192.168.80.105:9200"]
2

Hi @tsc036 and welcome :slight_smile:

It seems that the configuration you have is mixing input and general settings, but I cannot be sure as the indentation is lost, could you copy it as preformated text? (the preformatted text button in the toolbar can help with that </>)

Do you see any error in filebeat logs? Is filebeat started as root, or with enough privileges to read the files you are trying to collect?

3 
•type: log
   enabled: true
   paths:
     - /var/log/demisto/*.log
 logging.level: debug
 logging.to_files: true
 logging.files:
   path: /home/cloud-user/filebeat/filebeat.log
   name: filebeat
   keepfiles: 10
   permissions: 0644
 output.console:
   pretty:true
 output.logstash:
   hosts: ["192.168.80.105:9200"]

I think I have permissions because I'm not getting permissions denied error but filebeat will not start harvesting the file, but it will set it as the input.

4

Your input definition should be under filebeat.inputs, so your configuration should be something like:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/demisto/*.log

logging.level: debug
logging.to_files: true
logging.files:
  path: /home/cloud-user/filebeat/filebeat.log
  name: filebeat
  keepfiles: 10
  permissions: 0644

output.console:
  pretty:true

output.logstash:
  hosts: ["192.168.80.105:9200"]

And on this configuration you have two outputs enabled, but only one can be enabled at the same time.

In the logs you should see something about Harvesters starting for files under /var/log/demisto/.

5 
logging.level: debug
filebeat.inputs:
- type: log
  paths:
    - /var/log/demisto/*
output.logstash:
  hosts: ["192.168.80.105:9200"]

I have the filebeat.input I just missed it when I did copy past. For some reason the harvester starting doesn't happen

6

Are you running filebeat as root? Could you share the permissions of /var/log/demisto directory and the ones of one of the log files inside?

7

var/log/demisto drwxrwxrwx. 2 root root 4096 Aug 6 06:50 demisto

-rw-r--r--. 1 demisto demisto 4840 Jul 30 10:37 bolt_stats.log
-rw-r--r--. 1 demisto demisto 439 Jul 30 10:37 confdb.log
-rw-r--r--. 1 demisto demisto 570 Jul 30 10:37 conf.log
-rw-r--r--. 1 demisto demisto 243 Jul 30 10:37 confserver.log
-rwxrwxrwx. 1 demisto demisto 518542 Jul 26 16:45 content.log
-rw-r--r--. 1 demisto demisto 115488 Jul 30 10:38 dmesg.log
-rw-r--r--. 1 demisto demisto 11429 Jul 30 10:38 docker.log
-rw-r--r--. 1 demisto demisto 661 Jul 30 10:37 env.log
-rw-r--r--. 1 demisto demisto 247277 Jul 30 10:38 filesystem.log
-rw-r--r--. 1 demisto demisto 64135 Jul 30 10:37 go_stats.log
-rw-r--r--. 1 demisto demisto 26 Jul 30 10:38 journalctl.log
-rw-r--r--. 1 demisto demisto 314 Jul 30 10:37 license_data.log
-rw-r--r--. 1 demisto demisto 25 Jul 30 10:38 ml.log
-rw-r--r--. 1 demisto demisto 22319 Jul 30 10:37 network.log
-rw-r--r--. 1 demisto demisto 593 Jul 30 10:37 os.log
-rw-r--r--. 1 demisto demisto 0 Jul 30 10:38 preprocessRules.log
-rw-r--r--. 1 demisto demisto 13138 Jul 30 10:38 processes.log
-rwxr-xr-x. 1 demisto demisto 10485628 Aug 5 02:36 server-2019-08-05T06-36-39.344.log
-rwxr-xr-x. 1 demisto demisto 10485680 Aug 5 16:39 server-2019-08-05T20-39-39.392.log
-rwxr-xr-x. 1 demisto demisto 10485574 Aug 6 06:50 server-2019-08-06T10-50-39.522.log
-rwxr-xr-x. 1 demisto demisto 2290711 Aug 6 09:54 server.log
-rw-r--r--. 1 demisto demisto 63 Jul 30 10:38 version_control.log
-rw-r--r--. 1 demisto demisto 167440 Jul 30 10:38 web-app.log
-rw-r--r--. 1 demisto demisto 93 Jul 30 10:38 workers.log

8

From what I see here, this should work, I think we are missing something :thinking:
Is there any other error you can see in the logs?

Also, is logstash listening on 192.168.80.105:9200? 9200 is the default port of Elasticsearch.

9

I have elasticsearch forwarded to a different port on the instance. Im not getting any errors the harvester just isn't starting, is there anyway to get more info other than turning debugging on. Also, will filebeat not start if the logs don't fit a certain format?

10

i am wondering if the configuration is not being loaded, do you see something like this in logs when starting filebeat?

2019-08-08T12:01:22.448+0200	INFO	crawler/crawler.go:72	Loading Inputs: 1
2019-08-08T12:01:22.448+0200	INFO	log/input.go:152	Configured paths: [/var/log/demisto/*]
2019-08-08T12:01:22.448+0200	INFO	input/input.go:114	Starting input of type: log; ID: 1870334480389483564 
2019-08-08T12:01:22.448+0200	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.