# Not forwarding filebeat to logstash and no filebeat-\* index in kibana
**URL:** https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223
**Category:** Beats
**Tags:** filebeat
**Created:** [January 30, 2017, 2:42pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223 "2017-01-30T14:42:06Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@felixon](https://discuss.elastic.co/u/felixon)
#### Post date: [January 30, 2017, 2:42pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/1 "2017-01-30T14:42:06Z")
Hi everybody,
i installed elk stack today and configured everything.. to cut long story short: **filebeat not forwarding logs as i see it**
i have 3 servers:
xx.xx.xx.233 - **elasticsearch** v 5.1.2 -
> [elk@elasticsearch ~]$ sudo curl -X GET '[http://xx.xx.xx.233:9200](http://xx.xx.xx.233:9200)' {
> "name" : "elastic1",
> "cluster\_name" : "elasticluster",
> "cluster\_uuid" : "hffEWmKbTzKRerPawQUYoQ",
> "version" : {
> "number" : "5.1.2",
> "build\_hash" : "c8c4c16",
> "build\_date" : "2017-01-11T20:18:39.146Z",
> "build\_snapshot" : false,
> "lucene\_version" : "6.3.0"
> },
> "tagline" : "You Know, for Search"
> }
xx.xx.xx.232 - **kibana** v 5.1.2 - site is working
xx.xx.xx.231 - **logstash** v 5.1.2 + **filebeat** v 5.1.2 -
> [elk@logstash ~]$ sudo systemctl status logstash
> ● logstash.service - logstash
> Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
> Active: active (running) since Mon 2017-01-30 15:42:08 IST; 1min 23s ago
file beat status
> [elk@logstash ~]$ sudo systemctl status filebeat
> ● filebeat.service - filebeat
> Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
> Active: active (running) since Mon 2017-01-30 15:44:43 IST; 5s ago
filebeat yml
```
[elk@logstash ~]$ cat /etc/filebeat/filebeat.yml
- input_type: log
paths:
- /var/log/messages
- /var/log/secure
output.logstash:
hosts: ["xx.xx.xx.231:5044"]
ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
```
logstash conf
```
[elk@logstash conf.d]$ cat 01-beats-filter.conf
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => ["received_at", "%{@timestamp}"]
add_field => ["received_from", "%{host}"]
}
syslog_pri { }
date {
match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
}
}
}
```
input
```
[elk@logstash conf.d]$ cat 01-beats-input.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
```
output
```
[elk@logstash conf.d]$ cat 01-beats-output.conf
output {
elasticsearch {
hosts => ["xx.xx.xx.233:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
```
and here is the problem ☹ :
and heres a config test:
```
> [elk@logstash bin]$ sudo ./filebeat -e -c /etc/filebeat/filebeat.yml
> 2017/01/30 14:26:26.456427 beat.go:267: INFO Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]
> 2017/01/30 14:26:26.456457 beat.go:177: INFO Setup Beat: filebeat; Version: 5.1.2
> 2017/01/30 14:26:26.456601 logp.go:219: INFO Metrics logging every 30s
> 2017/01/30 14:26:26.457359 logstash.go:90: INFO Max Retries set to: 3
> 2017/01/30 14:26:26.457422 outputs.go:106: INFO Activated logstash as output plugin.
> 2017/01/30 14:26:26.457536 publish.go:291: INFO Publisher name: logstash
> 2017/01/30 14:26:26.457747 async.go:63: INFO Flush Interval set to: 1s
> 2017/01/30 14:26:26.457759 async.go:64: INFO Max Bulk Size set to: 2048
> 2017/01/30 14:26:26.457894 beat.go:207: INFO filebeat start running.
> 2017/01/30 14:26:26.458078 registrar.go:85: INFO Registry file set to: /usr/share/filebeat/bin/data/registry
> 2017/01/30 14:26:26.458106 registrar.go:106: INFO Loading registrar data from /usr/share/filebeat/bin/data/registry
> 2017/01/30 14:26:26.479380 registrar.go:123: INFO States Loaded from registrar: 0
> 2017/01/30 14:26:26.479433 crawler.go:34: INFO Loading Prospectors: 1
> 2017/01/30 14:26:26.479484 registrar.go:236: INFO Starting Registrar
> 2017/01/30 14:26:26.479485 prospector_log.go:57: INFO Prospector with previous states loaded: 0
> 2017/01/30 14:26:26.479554 sync.go:41: INFO Start sending events to output
> 2017/01/30 14:26:26.479595 crawler.go:46: INFO Loading Prospectors completed. Number of prospectors: 1
> 2017/01/30 14:26:26.479609 crawler.go:61: INFO All prospectors are initialised and running with 0 states to persist
> 2017/01/30 14:26:26.479631 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
> 2017/01/30 14:26:26.479660 prospector.go:111: INFO Starting prospector of type: log
> 2017/01/30 14:26:26.480251 log.go:84: INFO Harvester started for file: /var/log/secure
> 2017/01/30 14:26:26.480253 log.go:84: INFO Harvester started for file: /var/log/messages
> 2017/01/30 14:26:56.456824 logp.go:230: INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=2046 libbeat.logstash.publish.write_bytes=132 filebeat.harvester.running=2 filebeat.harvester.started=2 filebeat.harvester.open_files=2
> 2017/01/30 14:26:56.502738 single.go:140: ERR Connecting error publishing events (retrying): read tcp 172.16.50.231:35986->172.16.50.231:5044: i/o timeout
```
**i was looking for hours on guides and tried different configs but with no luck.. hope you could help me ?**
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 31, 2017, 3:17am UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/2 "2017-01-31T03:17:09Z")
Is there anything in ES? Use the `_cat` APIs to check.
---
### Author:  [@felixon](https://discuss.elastic.co/u/felixon)
#### Post date: [January 31, 2017, 7:59am UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/3 "2017-01-31T07:59:36Z")
hi, its working now. for others who have the same problem:
i installed logstash-beats-plugin
i enabled worker: 1 in the yml file
but: it seemes i can only get 1010 hits in the index
the events stoped flowing and all services are active and runing
can it possible be connected to some yml configuration?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [January 31, 2017, 12:43pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/4 "2017-01-31T12:43:54Z")
There seems to be a connection problem between filebeat and Logstash:
```auto
> 2017/01/30 14:26:56.502738 single.go:140: ERR Connecting error publishing events (retrying): read tcp 172.16.50.231:35986->172.16.50.231:5044: i/o timeout
```
You should investigate that further.
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [February 1, 2017, 12:52pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/5 "2017-02-01T12:52:49Z")
`logstash-input-beats` plugin version? btw. 5.2 version has been released recently, this might have a newer version of the input plugin.
Filebeat has a default read timeout of 30 seconds + the input plugin is supposed to send a keep-alive every 5 seconds for the time a batch is being processed. That is, for whatever reason either logstash is not sending the keep-alive (or final ACK), or the keep alive signal never makes it back to filebeat, or the connection has been silently dropped. Reason you see the first few events only is, data has been send and processed by logstash, but logstash did not properly ACK the events being processed to filebeat. Upon restart, filebeat will have to send the same events again.
Also check the logstash logs (in debug mode if possible). Besides potential issues with your network setup, I wonder if LS is overly busy (e.g. forcing a long gc cycle) or facing some problems with it's output.
---
### Author:  [@felixon](https://discuss.elastic.co/u/felixon)
#### Post date: [February 5, 2017, 8:31am UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/6 "2017-02-05T08:31:55Z")
hey,
after reviewing all of the above, i found out all my configtest was OK and all my services running properly.
my servers has:
16 G RAM
8 CPU
150 GB Storage
but still i see this:
all of my plugins are updated.
i this it has somthing to do wirh the logstash/file beat configuration.
**it has only sent logs on the first day of installation**
any ideas?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [February 6, 2017, 12:45pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/7 "2017-02-06T12:45:56Z")
Do you still see the output issues in the filebeat log?
---
### Author:  [@felixon](https://discuss.elastic.co/u/felixon)
#### Post date: [February 6, 2017, 12:59pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/8 "2017-02-06T12:59:07Z")
yes, im afraid all of my config is correct tho still no logs is flowing as my last ,msg above
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [February 6, 2017, 1:18pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/9 "2017-02-06T13:18:07Z")
Can you share the log file?
---
### Author:  [@felixon](https://discuss.elastic.co/u/felixon)
#### Post date: [February 6, 2017, 3:17pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/10 "2017-02-06T15:17:19Z")
yes, what logs would you like me to send?
---
### Author:  [@felixon](https://discuss.elastic.co/u/felixon)
#### Post date: [February 7, 2017, 12:33pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/11 "2017-02-07T12:33:57Z")
here some logs meanwhile: /var/log/filebeat/filebeat.log
```
2017-02-07T11:16:33+02:00 INFO No non-zero metrics in the last 30s
2017-02-07T11:17:03+02:00 INFO No non-zero metrics in the last 30s
2017-02-07T11:17:06+02:00 ERR Connecting error publishing events (retrying): dial tcp 172.16.50.231:5044: getsockopt: connection refused
2017-02-07T11:17:33+02:00 INFO No non-zero metrics in the last 30s
2017-02-07T11:18:03+02:00 INFO No non-zero metrics in the last 30s
2017-02-07T11:18:06+02:00 ERR Connecting error publishing events (retrying): dial tcp 172.16.50.231:5044: getsockopt: connection refused
2017-02-07T11:18:33+02:00 INFO No non-zero metrics in the last 30s
2017-02-07T11:19:03+02:00 INFO No non-zero metrics in the last 30s
2017-02-07T11:19:06+02:00 ERR Connecting error publishing events (retrying): dial tcp 172.16.50.231:5044: getsockopt: connection refused
2017-02-07T11:19:33+02:00 INFO No non-zero metrics in the last 30s
```
curl -v --cacert logstash-forwarder.crt https://elastic\_host\_ip:5044
```
* About to connect() to 172.16.50.233 port 5044 (#0)
* Trying 172.16.50.233...
* Connection refused
* Failed connect to 172.16.50.233:5044; Connection refused
* Closing connection 0
curl: (7) Failed connect to 172.16.50.233:5044; Connection refused
```
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [February 7, 2017, 3:38pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/12 "2017-02-07T15:38:40Z")
That are the ones I was looking for. It seems you still have the connection error. So that is the reason no data ends up in LS / ES.
There are quite a few forum posts with the same problem. Check out these posts to see if they help you solve the problem.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 7, 2017, 3:39pm UTC](https://discuss.elastic.co/t/not-forwarding-filebeat-to-logstash-and-no-filebeat-index-in-kibana/73223/13 "2017-03-07T15:39:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.