# Filebeat Is not Sending Logs to Logstash even Harvesting Successfull
**URL:** https://discuss.elastic.co/t/filebeat-is-not-sending-logs-to-logstash-even-harvesting-successfull/216579
**Category:** Beats
**Tags:** filebeat
**Created:** [January 26, 2020, 7:46pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-logs-to-logstash-even-harvesting-successfull/216579 "2020-01-26T19:46:36Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@mgnfcnt](https://discuss.elastic.co/u/mgnfcnt)
#### Post date: [January 26, 2020, 7:46pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-logs-to-logstash-even-harvesting-successfull/216579/1 "2020-01-26T19:46:36Z")
I am trying to use ELK space to collect file logs. Everything is OK untill filebeat integration. I can send the logs over tcp to logstash and see in the kibana.
But I couldn't achieved filebeat setup to send the logs. It seams sending the data but can not see at elasticsearch.
Using this command to create elasticsearch.
`docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name elasticsearch docker.elastic.co/elasticsearch/elasticsearch:7.5.2`
Kibana:
`docker run -d -p 5601:5601 -h kibana --name kibana --link elasticsearch:elasticsearch docker.elastic.co/kibana/kibana:7.5.2`
Logstash:
`docker run -d -p 5044:5044 -p 5000:5000 -h logstash --name logstash --link elasticsearch:elasticsearch -v c:/elk2/config-dir:/config-dir docker.elastic.co/logstash/logstash:7.5.2 -f /config-dir/logstash.conf`
Logstash.conf file
```
input {
beats {
type => "test"
port => "5044"
}
}
filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
if [message] =~ "\tat" {
grok {
match => ["message", "^(\tat)"]
add_tag => ["stacktrace"]
}
}
}
output {
stdout {
codec => rubydebug
}
# Sending properly parsed log events to elasticsearch
elasticsearch {
hosts => ["elasticsearch:9200"]
}
}
```
Running filebeat on windows 10 machine. Downloaded zip and filebeat.yml config
```
filebeat.modules:
filebeat.inputs:
- type: log
enabled: true
paths:
- C:/elk2/filebeat/log/*.log
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
output:
logstash:
hosts: ["localhost:5044"]
#Also tried 127.0.0.1/logstash/ip... as hosts here
```
Running powershell as admin mode and first
`./install-service-filebeat.ps1`
then
`./filebeat.exe -c ./filebeat.yml`
```
2020-01-26T22:28:45.652+0300 INFO log/harvester.go:251 Harvester started for file: C:\elk2\filebeat\log\logstash-mehmet.log
2020-01-26T22:29:15.651+0300 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":250,"time":{"ms":250}},"total":{"ticks":343,"time":{"ms":343},"value":343},"user":{"ticks":93,"time":{"ms":93}}},"handles":{"open":664},"info":{"ephemeral_id":"46f26124-44e5-4733-a259-4bed65d07a05","uptime":{"ms":32977}},"memstats":{"gc_next":9518416,"memory_alloc":6349856,"memory_total":10791408,"rss":39120896},"runtime":{"goroutines":28}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":8}}}}}
```
But no result at elastic search.
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [January 27, 2020, 8:58am UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-logs-to-logstash-even-harvesting-successfull/216579/2 "2020-01-27T08:58:43Z")
Hey!
As you mentioned, maybe Filebeat cannot parse the logs. In order to verify that you should just focus on Filebeat's part:
1. For debugging this use `console.output` instead of sending to Logstash -\> [https://www.elastic.co/guide/en/beats/filebeat/current/console-output.html](https://www.elastic.co/guide/en/beats/filebeat/current/console-output.html)
2. I would suggest to first try with a bare minimum config like [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log) (without multiline settings) and later try to add them.
Could you check if there are any errors reported in Filebeat's output while running with the above steps? (also you can provide the output here)
Thank you!
---
### Author:  [@mgnfcnt](https://discuss.elastic.co/u/mgnfcnt)
#### Post date: [January 27, 2020, 9:05am UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-logs-to-logstash-even-harvesting-successfull/216579/3 "2020-01-27T09:05:33Z")
Hey @ChrsMark It is not consistent for now, when first start it is sending sometimes, but not mostly here are my latest settings:
filebeat.yml
```
filebeat:
prospectors:
-
paths:
- C:/elk2/logs/*.log
input_type: log
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
output:
logstash:
enabled: true
hosts: ["localhost:5044"]
logging.level: DEBUG
logging.selectors: ["*"]
```
logstash.conf
```
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
}
}
```
logstash docker command:
`docker run -d -p "5044:5044/tcp" -p 5000:5000 -p 5144:5144 -p 5244:5244 -h logstash --name logstash --link elasticsearch:elasticsearch -v c:/elk2/config-dir:/config-dir docker.elastic.co/logstash/logstash:6.5.1 -f /config-dir/logstash.conf`
Logs from filebeat:
```
2020-01-27T12:07:01.117+0300 DEBUG [input] log/input.go:174 Start next scan
2020-01-27T12:07:01.119+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\x.log
2020-01-27T12:07:01.119+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\x.log, offset: 100
2020-01-27T12:07:01.119+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\x.log
2020-01-27T12:07:01.119+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\y.log
2020-01-27T12:07:01.119+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\y.log, offset: 0
2020-01-27T12:07:01.119+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\y.log
2020-01-27T12:07:01.119+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\z.log
2020-01-27T12:07:01.120+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\z.log, offset: 0
2020-01-27T12:07:01.120+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\z.log
2020-01-27T12:07:01.120+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\hakan.log
2020-01-27T12:07:01.120+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\hakan.log, offset: 0
2020-01-27T12:07:01.120+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\hakan.log
2020-01-27T12:07:01.120+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\sample.log
2020-01-27T12:07:01.121+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\sample.log, offset: 0
2020-01-27T12:07:01.121+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\sample.log
2020-01-27T12:07:01.121+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\t.log
2020-01-27T12:07:01.121+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\t.log, offset: 0
2020-01-27T12:07:01.121+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\t.log
2020-01-27T12:07:01.121+0300 DEBUG [input] log/input.go:195 input states cleaned up. Before: 6, After: 6, Pending: 0
2020-01-27T12:07:05.812+0300 DEBUG [multiline] multiline/multiline.go:146 Multiline event flushed because timeout reached.
2020-01-27T12:07:05.812+0300 INFO log/harvester.go:279 File is inactive: C:\elk2\logs\t.log. Closing because close_inactive of 5m0s reached.
2020-01-27T12:07:05.812+0300 DEBUG [harvester] log/harvester.go:510 Stopping harvester for file: C:\elk2\logs\t.log
2020-01-27T12:07:05.813+0300 DEBUG [harvester] log/harvester.go:520 Closing file: C:\elk2\logs\t.log
2020-01-27T12:07:05.813+0300 DEBUG [harvester] log/harvester.go:390 Update state: C:\elk2\logs\t.log, offset: 0
2020-01-27T12:07:05.813+0300 DEBUG [harvester] log/harvester.go:531 harvester cleanup finished for file: C:\elk2\logs\t.log
2020-01-27T12:07:05.813+0300 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2020-01-27T12:07:05.813+0300 DEBUG [registrar] registrar/registrar.go:345 Processing 1 events
2020-01-27T12:07:05.813+0300 DEBUG [registrar] registrar/registrar.go:315 Registrar state updates processed. Count: 1
2020-01-27T12:07:05.813+0300 DEBUG [registrar] registrar/registrar.go:335 Registrar states cleaned up. Before: 6, After: 6, Pending: 0
2020-01-27T12:07:05.813+0300 DEBUG [registrar] registrar/registrar.go:400 Write registry file: C:\elk2\filebeat-6.5.1-windows-x86_64\data\registry
2020-01-27T12:07:05.833+0300 DEBUG [registrar] registrar/registrar.go:393 Registry file updated. 6 states written.
2020-01-27T12:07:05.844+0300 DEBUG [harvester] log/log.go:102 End of file reached: C:\elk2\logs\hakan.log; Backoff now.
2020-01-27T12:07:05.851+0300 DEBUG [harvester] log/log.go:102 End of file reached: C:\elk2\logs\sample.log; Backoff now.
2020-01-27T12:07:05.872+0300 DEBUG [harvester] log/log.go:102 End of file reached: C:\elk2\logs\x.log; Backoff now.
2020-01-27T12:07:06.031+0300 DEBUG [harvester] log/log.go:102 End of file reached: C:\elk2\logs\y.log; Backoff now.
2020-01-27T12:07:06.080+0300 DEBUG [harvester] log/log.go:102 End of file reached: C:\elk2\logs\z.log; Backoff now.
2020-01-27T12:07:11.124+0300 DEBUG [input] input/input.go:152 Run input
2020-01-27T12:07:11.124+0300 DEBUG [input] log/input.go:174 Start next scan
2020-01-27T12:07:11.129+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\sample.log
2020-01-27T12:07:11.129+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\sample.log, offset: 0
2020-01-27T12:07:11.130+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\sample.log
2020-01-27T12:07:11.130+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\t.log
2020-01-27T12:07:11.130+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\t.log, offset: 0
2020-01-27T12:07:11.131+0300 DEBUG [input] log/input.go:503 Resuming harvesting of file: C:\elk2\logs\t.log, offset: 0, new size: 40
2020-01-27T12:07:11.132+0300 DEBUG [harvester] log/harvester.go:489 Setting offset for file based on seek: C:\elk2\logs\t.log
2020-01-27T12:07:11.132+0300 DEBUG [harvester] log/harvester.go:475 Setting offset for file: C:\elk2\logs\t.log. Offset: 0
2020-01-27T12:07:11.132+0300 DEBUG [harvester] log/harvester.go:390 Update state: C:\elk2\logs\t.log, offset: 0
2020-01-27T12:07:11.132+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\x.log
2020-01-27T12:07:11.132+0300 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2020-01-27T12:07:11.132+0300 DEBUG [registrar] registrar/registrar.go:345 Processing 1 events
2020-01-27T12:07:11.132+0300 INFO log/harvester.go:254 Harvester started for file: C:\elk2\logs\t.log
2020-01-27T12:07:11.132+0300 DEBUG [registrar] registrar/registrar.go:315 Registrar state updates processed. Count: 1
2020-01-27T12:07:11.133+0300 DEBUG [registrar] registrar/registrar.go:400 Write registry file: C:\elk2\filebeat-6.5.1-windows-x86_64\data\registry
2020-01-27T12:07:11.133+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\x.log, offset: 100
2020-01-27T12:07:11.133+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\x.log
2020-01-27T12:07:11.133+0300 DEBUG [harvester] log/log.go:102 End of file reached: C:\elk2\logs\t.log; Backoff now.
2020-01-27T12:07:11.134+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\y.log
2020-01-27T12:07:11.136+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\y.log, offset: 0
2020-01-27T12:07:11.136+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\y.log
2020-01-27T12:07:11.136+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\z.log
2020-01-27T12:07:11.137+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\z.log, offset: 0
2020-01-27T12:07:11.137+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\z.log
2020-01-27T12:07:11.137+0300 DEBUG [input] log/input.go:404 Check file for harvesting: C:\elk2\logs\hakan.log
2020-01-27T12:07:11.138+0300 DEBUG [input] log/input.go:494 Update existing file for harvesting: C:\elk2\logs\hakan.log, offset: 0
2020-01-27T12:07:11.138+0300 DEBUG [input] log/input.go:546 Harvester for file is still running: C:\elk2\logs\hakan.log
2020-01-27T12:07:11.138+0300 DEBUG [input] log/input.go:195 input states cleaned up. Before: 6, After: 6, Pending: 0
2020-01-27T12:07:11.169+0300 DEBUG [registrar] registrar/registrar.go:393 Registry file updated. 6 states written.
2020-01-27T12:07:12.135+0300 DEBUG [harvester] log/log.go:102 End of file reached: C:\elk2\logs\t.log; Backoff now.
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 24, 2020, 9:10am UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-logs-to-logstash-even-harvesting-successfull/216579/4 "2020-02-24T09:10:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.