Filebeat Is not Sending Logs to Logstash even Harvesting Successfull

mgnfcnt · January 26, 2020, 7:46pm

I am trying to use ELK space to collect file logs. Everything is OK untill filebeat integration. I can send the logs over tcp to logstash and see in the kibana.
But I couldn't achieved filebeat setup to send the logs. It seams sending the data but can not see at elasticsearch.

Using this command to create elasticsearch.

docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name elasticsearch docker.elastic.co/elasticsearch/elasticsearch:7.5.2

Kibana:

docker run -d -p 5601:5601 -h kibana --name kibana --link elasticsearch:elasticsearch docker.elastic.co/kibana/kibana:7.5.2

Logstash:

docker run -d -p 5044:5044 -p 5000:5000 -h logstash --name logstash --link elasticsearch:elasticsearch -v c:/elk2/config-dir:/config-dir docker.elastic.co/logstash/logstash:7.5.2 -f /config-dir/logstash.conf

Logstash.conf file

input {
    beats {
	    type => "test"
        port => "5044"
    }
}
 
filter {
  #If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }
 
}
 
output {
   
  stdout {
    codec => rubydebug
  }
 
  # Sending properly parsed log events to elasticsearch
  elasticsearch {
    hosts => ["elasticsearch:9200"]
  }
}

Running filebeat on windows 10 machine. Downloaded zip and filebeat.yml config

filebeat.modules:
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:/elk2/filebeat/log/*.log
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  
output:
  logstash:
    hosts: ["localhost:5044"]
#Also tried 127.0.0.1/logstash/ip... as hosts here

Running powershell as admin mode and first

./install-service-filebeat.ps1

then

./filebeat.exe -c ./filebeat.yml

2020-01-26T22:28:45.652+0300    INFO    log/harvester.go:251    Harvester started for file: C:\elk2\filebeat\log\logstash-mehmet.log
2020-01-26T22:29:15.651+0300    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":250,"time":{"ms":250}},"total":{"ticks":343,"time":{"ms":343},"value":343},"user":{"ticks":93,"time":{"ms":93}}},"handles":{"open":664},"info":{"ephemeral_id":"46f26124-44e5-4733-a259-4bed65d07a05","uptime":{"ms":32977}},"memstats":{"gc_next":9518416,"memory_alloc":6349856,"memory_total":10791408,"rss":39120896},"runtime":{"goroutines":28}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":8}}}}}

But no result at elastic search.

ChrsMark · January 27, 2020, 8:58am

Hey!

As you mentioned, maybe Filebeat cannot parse the logs. In order to verify that you should just focus on Filebeat's part:

For debugging this use console.output instead of sending to Logstash -> https://www.elastic.co/guide/en/beats/filebeat/current/console-output.html
I would suggest to first try with a bare minimum config like https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log (without multiline settings) and later try to add them.

Could you check if there are any errors reported in Filebeat's output while running with the above steps? (also you can provide the output here)

Thank you!

mgnfcnt · January 27, 2020, 9:05am

Hey @ChrsMark It is not consistent for now, when first start it is sending sometimes, but not mostly here are my latest settings:

filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - C:/elk2/logs/*.log
      input_type: log
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
      
output:
  logstash:
    enabled: true
    hosts: ["localhost:5044"]

logging.level: DEBUG
logging.selectors: ["*"]

logstash.conf

input {
    beats {
        port => 5044
    }
}
 
output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
  }
}

logstash docker command:

docker run -d -p "5044:5044/tcp" -p 5000:5000 -p 5144:5144 -p 5244:5244 -h logstash --name logstash --link elasticsearch:elasticsearch -v c:/elk2/config-dir:/config-dir docker.elastic.co/logstash/logstash:6.5.1 -f /config-dir/logstash.conf

Logs from filebeat:

2020-01-27T12:07:01.117+0300    DEBUG   [input] log/input.go:174        Start next scan
2020-01-27T12:07:01.119+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\x.log
2020-01-27T12:07:01.119+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\x.log, offset: 100
2020-01-27T12:07:01.119+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\x.log
2020-01-27T12:07:01.119+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\y.log
2020-01-27T12:07:01.119+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\y.log, offset: 0
2020-01-27T12:07:01.119+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\y.log
2020-01-27T12:07:01.119+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\z.log
2020-01-27T12:07:01.120+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\z.log, offset: 0
2020-01-27T12:07:01.120+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\z.log
2020-01-27T12:07:01.120+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\hakan.log
2020-01-27T12:07:01.120+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\hakan.log, offset: 0
2020-01-27T12:07:01.120+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\hakan.log
2020-01-27T12:07:01.120+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\sample.log
2020-01-27T12:07:01.121+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\sample.log, offset: 0
2020-01-27T12:07:01.121+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\sample.log
2020-01-27T12:07:01.121+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\t.log
2020-01-27T12:07:01.121+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\t.log, offset: 0
2020-01-27T12:07:01.121+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\t.log
2020-01-27T12:07:01.121+0300    DEBUG   [input] log/input.go:195        input states cleaned up. Before: 6, After: 6, Pending: 0
2020-01-27T12:07:05.812+0300    DEBUG   [multiline]     multiline/multiline.go:146      Multiline event flushed because timeout reached.
2020-01-27T12:07:05.812+0300    INFO    log/harvester.go:279    File is inactive: C:\elk2\logs\t.log. Closing because close_inactive of 5m0s reached.
2020-01-27T12:07:05.812+0300    DEBUG   [harvester]     log/harvester.go:510    Stopping harvester for file: C:\elk2\logs\t.log
2020-01-27T12:07:05.813+0300    DEBUG   [harvester]     log/harvester.go:520    Closing file: C:\elk2\logs\t.log
2020-01-27T12:07:05.813+0300    DEBUG   [harvester]     log/harvester.go:390    Update state: C:\elk2\logs\t.log, offset: 0
2020-01-27T12:07:05.813+0300    DEBUG   [harvester]     log/harvester.go:531    harvester cleanup finished for file: C:\elk2\logs\t.log
2020-01-27T12:07:05.813+0300    DEBUG   [acker] beater/acker.go:64      stateful ack    {"count": 1}
2020-01-27T12:07:05.813+0300    DEBUG   [registrar]     registrar/registrar.go:345      Processing 1 events
2020-01-27T12:07:05.813+0300    DEBUG   [registrar]     registrar/registrar.go:315      Registrar state updates processed. Count: 1
2020-01-27T12:07:05.813+0300    DEBUG   [registrar]     registrar/registrar.go:335      Registrar states cleaned up. Before: 6, After: 6, Pending: 0
2020-01-27T12:07:05.813+0300    DEBUG   [registrar]     registrar/registrar.go:400      Write registry file: C:\elk2\filebeat-6.5.1-windows-x86_64\data\registry
2020-01-27T12:07:05.833+0300    DEBUG   [registrar]     registrar/registrar.go:393      Registry file updated. 6 states written.
2020-01-27T12:07:05.844+0300    DEBUG   [harvester]     log/log.go:102  End of file reached: C:\elk2\logs\hakan.log; Backoff now.
2020-01-27T12:07:05.851+0300    DEBUG   [harvester]     log/log.go:102  End of file reached: C:\elk2\logs\sample.log; Backoff now.
2020-01-27T12:07:05.872+0300    DEBUG   [harvester]     log/log.go:102  End of file reached: C:\elk2\logs\x.log; Backoff now.
2020-01-27T12:07:06.031+0300    DEBUG   [harvester]     log/log.go:102  End of file reached: C:\elk2\logs\y.log; Backoff now.
2020-01-27T12:07:06.080+0300    DEBUG   [harvester]     log/log.go:102  End of file reached: C:\elk2\logs\z.log; Backoff now.
2020-01-27T12:07:11.124+0300    DEBUG   [input] input/input.go:152      Run input
2020-01-27T12:07:11.124+0300    DEBUG   [input] log/input.go:174        Start next scan
2020-01-27T12:07:11.129+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\sample.log
2020-01-27T12:07:11.129+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\sample.log, offset: 0
2020-01-27T12:07:11.130+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\sample.log
2020-01-27T12:07:11.130+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\t.log
2020-01-27T12:07:11.130+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\t.log, offset: 0
2020-01-27T12:07:11.131+0300    DEBUG   [input] log/input.go:503        Resuming harvesting of file: C:\elk2\logs\t.log, offset: 0, new size: 40
2020-01-27T12:07:11.132+0300    DEBUG   [harvester]     log/harvester.go:489    Setting offset for file based on seek: C:\elk2\logs\t.log
2020-01-27T12:07:11.132+0300    DEBUG   [harvester]     log/harvester.go:475    Setting offset for file: C:\elk2\logs\t.log. Offset: 0
2020-01-27T12:07:11.132+0300    DEBUG   [harvester]     log/harvester.go:390    Update state: C:\elk2\logs\t.log, offset: 0
2020-01-27T12:07:11.132+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\x.log
2020-01-27T12:07:11.132+0300    DEBUG   [acker] beater/acker.go:64      stateful ack    {"count": 1}
2020-01-27T12:07:11.132+0300    DEBUG   [registrar]     registrar/registrar.go:345      Processing 1 events
2020-01-27T12:07:11.132+0300    INFO    log/harvester.go:254    Harvester started for file: C:\elk2\logs\t.log
2020-01-27T12:07:11.132+0300    DEBUG   [registrar]     registrar/registrar.go:315      Registrar state updates processed. Count: 1
2020-01-27T12:07:11.133+0300    DEBUG   [registrar]     registrar/registrar.go:400      Write registry file: C:\elk2\filebeat-6.5.1-windows-x86_64\data\registry
2020-01-27T12:07:11.133+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\x.log, offset: 100
2020-01-27T12:07:11.133+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\x.log
2020-01-27T12:07:11.133+0300    DEBUG   [harvester]     log/log.go:102  End of file reached: C:\elk2\logs\t.log; Backoff now.
2020-01-27T12:07:11.134+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\y.log
2020-01-27T12:07:11.136+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\y.log, offset: 0
2020-01-27T12:07:11.136+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\y.log
2020-01-27T12:07:11.136+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\z.log
2020-01-27T12:07:11.137+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\z.log, offset: 0
2020-01-27T12:07:11.137+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\z.log
2020-01-27T12:07:11.137+0300    DEBUG   [input] log/input.go:404        Check file for harvesting: C:\elk2\logs\hakan.log
2020-01-27T12:07:11.138+0300    DEBUG   [input] log/input.go:494        Update existing file for harvesting: C:\elk2\logs\hakan.log, offset: 0
2020-01-27T12:07:11.138+0300    DEBUG   [input] log/input.go:546        Harvester for file is still running: C:\elk2\logs\hakan.log
2020-01-27T12:07:11.138+0300    DEBUG   [input] log/input.go:195        input states cleaned up. Before: 6, After: 6, Pending: 0
2020-01-27T12:07:11.169+0300    DEBUG   [registrar]     registrar/registrar.go:393      Registry file updated. 6 states written.
2020-01-27T12:07:12.135+0300    DEBUG   [harvester]     log/log.go:102  End of file reached: C:\elk2\logs\t.log; Backoff now.

system · February 24, 2020, 9:10am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat is not sending logs to Logstash Beats docker , filebeat	0	196	June 23, 2024
Trouble with sending logs from filebeat to elasticsearch Beats beats-module , filebeat	8	740	December 20, 2021
Filebeat in docker doesn't send logs to logstash in another docker Beats filebeat	6	2749	July 19, 2018
Filebeat fails to send docker container logs to Logstash Beats filebeat	4	3587	July 4, 2018
Filebeat is not sending logs to Elasticsearch Beats docker , filebeat	4	597	April 28, 2024

Filebeat Is not Sending Logs to Logstash even Harvesting Successfull

Related topics