Filebeat Output error while using file

Elastic Stack Beats
1

PS C:\Program Files\Filebeat> ./filebeat -e -c filebeat0.yml -d "publish"
2016/05/05 07:26:16.287762 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/05/05 07:26:16.287762 logstash.go:106: INFO Max Retries set to: 3
2016/05/05 07:26:16.292766 outputs.go:126: INFO Activated logstash as output plugin.
2016/05/05 07:26:16.292766 publish.go:232: DBG Create output worker
2016/05/05 07:26:16.292766 publish.go:274: DBG No output is defined to store the topology. The server fields might not
be filled.
2016/05/05 07:26:16.292766 publish.go:288: INFO Publisher name: akash
2016/05/05 07:26:16.297770 async.go:78: INFO Flush Interval set to: 1s
2016/05/05 07:26:16.297770 async.go:84: INFO Max Bulk Size set to: 2048
2016/05/05 07:26:16.297770 async.go:92: DBG create bulk processing worker (interval=1s, bulk size=2048)
2016/05/05 07:26:16.298770 beat.go:147: INFO Init Beat: filebeat; Version: 1.2.1
2016/05/05 07:26:16.298770 beat.go:173: INFO filebeat sucessfully setup. Start running.
2016/05/05 07:26:16.299771 registrar.go:66: INFO Registry file set to: C:\Program Files\Filebeat.filebeat
2016/05/05 07:26:16.299771 registrar.go:76: INFO Loading registrar data from C:\Program Files\Filebeat.filebeat
2016/05/05 07:26:16.332015 prospector.go:132: INFO Set ignore_older duration to 0
2016/05/05 07:26:16.332015 prospector.go:132: INFO Set close_older duration to 1h0m0s
2016/05/05 07:26:16.333016 prospector.go:132: INFO Set scan_frequency duration to 10s
2016/05/05 07:26:16.333016 prospector.go:89: INFO Invalid input type set:
2016/05/05 07:26:16.333016 prospector.go:92: INFO Input type set to: log
2016/05/05 07:26:16.333016 prospector.go:132: INFO Set backoff duration to 1s
2016/05/05 07:26:16.333016 prospector.go:132: INFO Set max_backoff duration to 10s
2016/05/05 07:26:16.333016 prospector.go:112: INFO force_close_file is disabled
2016/05/05 07:26:16.333016 prospector.go:132: INFO Set ignore_older duration to 0
2016/05/05 07:26:16.333016 prospector.go:132: INFO Set close_older duration to 1h0m0s
2016/05/05 07:26:16.334016 prospector.go:132: INFO Set scan_frequency duration to 10s
2016/05/05 07:26:16.334016 prospector.go:89: INFO Invalid input type set: err
2016/05/05 07:26:16.334016 prospector.go:92: INFO Input type set to: log
2016/05/05 07:26:16.334016 prospector.go:132: INFO Set backoff duration to 1s
2016/05/05 07:26:16.334016 prospector.go:132: INFO Set max_backoff duration to 10s
2016/05/05 07:26:16.334016 prospector.go:112: INFO force_close_file is disabled
2016/05/05 07:26:16.334016 prospector.go:142: INFO Starting prospector of type: log
2016/05/05 07:26:16.333016 spooler.go:77: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2016/05/05 07:26:16.335018 prospector.go:142: INFO Starting prospector of type: log
2016/05/05 07:26:16.356031 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial
\tutorial\20160402000001.err
2016/05/05 07:26:16.357032 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial
\tutorial\20160402000101.err
2016/05/05 07:26:16.358033 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial
\tutorial\20160402000201.err
2016/05/05 07:26:16.359033 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial
\tutorial\20160402000401.err
2016/05/05 07:26:16.359033 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial
\tutorial\20160402000301.err
2016/05/05 07:26:16.360034 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial
\tutorial\20160402000601.err
2016/05/05 07:26:16.360034 log.go:113: INFO Harvester started for file: c:\Users\akashsoni\Downloads\k\logstash-tutorial
\tutorial\20160402000502.err
2016/05/05 07:26:19.661256 crawler.go:78: INFO All prospectors initialised with 3064 states to persist
2016/05/05 07:26:19.663256 registrar.go:83: INFO Starting Registrar
2016/05/05 07:26:19.664257 publish.go:88: INFO Start sending events to output

This is my filebeat output : courser is blinking continuously but not showing output why ?
when i use same code as i written above in filebeat config file with .log extension it will work fine.
tomorrow this code is running for both err and log sup-rattly or together.

2

My filebeat0.yml
filebeat:

prospectors:

paths:

  • c:/Users/akashsoni/Downloads/k/logstash-tutorial/tutorial/*.log
  • c:/Users/akashsoni/Downloads/k/logstash-tutorial/tutorial/*.err

input_type: log

registry_file: "C:/ProgramData/filebeat/registry"

output:

logstash:
hosts: ["localhost:5044"]
shipper:

logging:

files:
rotateeverybytes: 10485760 # = 10MB

3

What sort of output do you expect?

4

Transfer of data that what filebeat is sending,and i used publish command.

5

You only have the logstash output, you may want to look at https://www.elastic.co/guide/en/beats/filebeat/current/console-output.html

6

shall i increase the size of buffer?

7

my logstash is also not giving output in elastic search.

8

Hi,

suppose I make one log file and send that file from filebeat to logstash and then to elasticsearch it works fine ,but when i remove the file from my elasticsearch ,then rerun with same file it will not show any output just my courser is blinking on filebeat side ,filebeat is ready to send but not send any data to logstash.Same problem with the logstash also.Why?

9

if you are going to resend the same file, you will need to remove the filebeat registry. it keeps track of what it already sent and will not send it again.

10

Thankyou sir