Hi,
I'm new to filebeat. I've read similar topics like this but none seems to solve my case.
I'm starting filebeat in the command line to debug it before running it as a service.
As it happens in the other topics like this, filebeat sends logs entries when it is started, but stops sending new entries thereafter.
First I tested it with a real application log, but as it didn´t work, I created a script to keep adding lines to a file while filebeat process it.
I´m testing output to console before logstash.
Folowing are the scripts and logs:
Script:
for i in {1..1000}; do echo -e "{ "test": "$(date)" }" >> /tmp/somaEAR-srvclnSOMAxx.log; sleep 10; done
Some lines of the log file:
{ "test": "Wed Apr 26 16:33:41 -03 2017" }
{ "test": "Wed Apr 26 16:33:51 -03 2017" }
{ "test": "Wed Apr 26 16:34:01 -03 2017" }
{ "test": "Wed Apr 26 16:34:11 -03 2017" }
{ "test": "Wed Apr 26 16:34:21 -03 2017" }
{ "test": "Wed Apr 26 16:34:31 -03 2017" }
{ "test": "Wed Apr 26 16:34:41 -03 2017" }
{ "test": "Wed Apr 26 16:34:51 -03 2017" }
{ "test": "Wed Apr 26 16:35:01 -03 2017" }
{ "test": "Wed Apr 26 16:35:11 -03 2017" }
{ "test": "Wed Apr 26 16:35:21 -03 2017" }
filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /tmp/somaEAR-srvclnSOMA*
document_type: somaear
output.console:
pretty: true
logging:
level: debug
to_syslog: false
to_files: true
files:
path: /var/log/filebeat
name: filebeat.log
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7
Log of filebeat running:
[f0113886@li1576 filebeat]$ sudo /usr/bin/filebeat.sh -e -v -d '*'
2017/04/26 18:44:42.545233 beat.go:285: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017/04/26 18:44:42.545281 beat.go:186: INFO Setup Beat: filebeat; Version: 5.3.0
2017/04/26 18:44:42.545334 outputs.go:108: INFO Activated console as output plugin.
2017/04/26 18:44:42.545425 publish.go:295: INFO Publisher name: li1576.portoseguro.brasil
2017/04/26 18:44:42.545580 async.go:63: INFO Flush Interval set to: 1s
2017/04/26 18:44:42.545604 async.go:64: INFO Max Bulk Size set to: 2048
2017/04/26 18:44:42.545936 metrics.go:23: INFO Metrics logging every 30s
2017/04/26 18:44:42.546099 beat.go:221: INFO filebeat start running.
2017/04/26 18:44:42.546141 registrar.go:85: INFO Registry file set to: /var/lib/filebeat/registry
2017/04/26 18:44:42.546177 registrar.go:106: INFO Loading registrar data from /var/lib/filebeat/registry
2017/04/26 18:44:42.546233 registrar.go:123: INFO States Loaded from registrar: 0
2017/04/26 18:44:42.546266 crawler.go:38: INFO Loading Prospectors: 1
2017/04/26 18:44:42.546396 prospector_log.go:61: INFO Prospector with previous states loaded: 0
2017/04/26 18:44:42.546503 prospector.go:124: INFO Starting prospector of type: log; id: 14707180048950103318
2017/04/26 18:44:42.546559 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017/04/26 18:44:42.546617 registrar.go:236: INFO Starting Registrar
2017/04/26 18:44:42.546685 sync.go:41: INFO Start sending events to output
2017/04/26 18:44:42.546767 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017/04/26 18:44:42.547236 log.go:91: INFO Harvester started for file: /tmp/somaEAR-srvclnSOMAxx.log
{
"@timestamp": "2017-04-26T18:44:42.547Z",
"beat": {
"hostname": "li1576.portoseguro.brasil",
"name": "li1576.portoseguro.brasil",
"version": "5.3.0"
},
"input_type": "log",
"message": "{ "test": "Wed Apr 26 15:44:29 -03 2017" }",
"offset": 43,
"source": "/tmp/somaEAR-srvclnSOMAxx.log",
"type": "somaear"
}
2017/04/26 18:44:47.548286 console.go:83: CRIT Unable to publish events to console:
{
"@timestamp": "2017-04-26T18:44:42.547Z",
"beat": {
"hostname": "li1576.portoseguro.brasil",
"name": "li1576.portoseguro.brasil",
"version": "5.3.0"
},
"input_type": "log",
"message": "{ "test": "Wed Apr 26 15:44:39 -03 2017" }",
"offset": 86,
"source": "/tmp/somaEAR-srvclnSOMAxx.log",
"type": "somaear"
}
2017/04/26 18:44:47.548393 console.go:83: CRIT Unable to publish events to console:
2017/04/26 18:45:12.546805 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.publisher.published_events=2
2017/04/26 18:45:42.547414 metrics.go:34: INFO No non-zero metrics in the last 30s
2017/04/26 18:46:12.547328 metrics.go:34: INFO No non-zero metrics in the last 30s
2017/04/26 18:46:42.546811 metrics.go:34: INFO No non-zero metrics in the last 30s
2017/04/26 18:47:12.547216 metrics.go:34: INFO No non-zero metrics in the last 30s
2017/04/26 18:47:42.546848 metrics.go:34: INFO No non-zero metrics in the last 30s
2017/04/26 18:48:12.547476 metrics.go:34: INFO No non-zero metrics in the last 30s
2017/04/26 18:48:42.547008 metrics.go:34: INFO No non-zero metrics in the last 30s