Running filebeat

(Sunil Chaudhari) #1

I am running filebeat using
PS C:\Program Files\Filebeat> Start-Service filebeat

However it just creates one file Start-Service in home directory and nothing happens.
Please help.
I am referring below link.


(Tudor Golubenco) #2

Did you run the install script? What's the output of Get-Service filebeat?

(Sunil Chaudhari) #3

Here is the status. its running. Where to see logs?

Status Name DisplayName

Running filebeat filebeat

(Sunil Chaudhari) #4

I changed loglevel to debug and I get this when I run filebeat.
I don't understand why its not sending contents to elasticsearch.

2016-04-19T16:45:49+05:30 DBG  Disable stderr logging
2016-04-19T16:45:49+05:30 DBG  Initializing output plugins
2016-04-19T16:45:49+05:30 INFO GeoIP disabled: No paths were set under output.geoip.paths
2016-04-19T16:45:49+05:30 DBG  ES Ping(url=http://localhost:9200, timeout=1m30s)
2016-04-19T16:45:50+05:30 DBG  Ping status code: 200
2016-04-19T16:45:50+05:30 INFO Activated elasticsearch as output plugin.
2016-04-19T16:45:50+05:30 DBG  Create output worker
2016-04-19T16:45:50+05:30 DBG  No output is defined to store the topology. The server fields might not be filled.
2016-04-19T16:45:50+05:30 INFO Publisher name: hostName
2016-04-19T16:45:50+05:30 INFO Flush Interval set to: 1s
2016-04-19T16:45:50+05:30 INFO Max Bulk Size set to: 50
2016-04-19T16:45:50+05:30 DBG  create bulk processing worker (interval=1s, bulk size=50)
2016-04-19T16:45:50+05:30 INFO Init Beat: filebeat; Version: 1.2.1
2016-04-19T16:45:50+05:30 INFO filebeat sucessfully setup. Start running.
2016-04-19T16:45:50+05:30 INFO Registry file set to: C:\ProgramData\filebeat\registry
2016-04-19T16:45:50+05:30 INFO Loading registrar data from C:\ProgramData\filebeat\registry
2016-04-19T16:45:50+05:30 DBG  Set idleTimeoutDuration to 5s
2016-04-19T16:45:50+05:30 DBG  File Configs: [C:\var\log\test.log]
2016-04-19T16:45:50+05:30 INFO Set ignore_older duration to 0
2016-04-19T16:45:50+05:30 INFO Set close_older duration to 1h0m0s
2016-04-19T16:45:50+05:30 INFO Set scan_frequency duration to 10s
2016-04-19T16:45:50+05:30 INFO Input type set to: log
2016-04-19T16:45:50+05:30 INFO Set backoff duration to 1s
2016-04-19T16:45:50+05:30 INFO Set max_backoff duration to 10s
2016-04-19T16:45:50+05:30 INFO force_close_file is disabled
2016-04-19T16:45:50+05:30 DBG  Waiting for 1 prospectors to initialise
2016-04-19T16:45:50+05:30 INFO Starting prospector of type: log
2016-04-19T16:45:50+05:30 DBG  exclude_files: []
2016-04-19T16:45:50+05:30 DBG  scan path C:\var\log\test.log
2016-04-19T16:45:50+05:30 DBG  scan path C:\var\log\test.log
2016-04-19T16:45:50+05:30 DBG  No pending prospectors. Finishing setup
2016-04-19T16:45:50+05:30 INFO All prospectors initialised with 0 states to persist
2016-04-19T16:45:50+05:30 INFO Starting Registrar
2016-04-19T16:45:50+05:30 INFO Start sending events to output
2016-04-19T16:45:50+05:30 DBG  Windows is interactive: true
2016-04-19T16:45:50+05:30 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2016-04-19T16:45:52+05:30 DBG  Flushing spooler because of timeout. Events flushed: 0
2016-04-19T16:46:00+05:30 DBG  Start next scan
2016-04-19T16:46:00+05:30 DBG  scan path C:\var\log\test.log
2016-04-19T16:46:00+05:30 DBG  Flushing spooler because of timeout. Events flushed: 0
2016-04-19T16:46:07+05:30 DBG  Flushing spooler because of timeout. Events flushed: 0

(Tudor Golubenco) #5

Can you post your configuration file as well, please?

(Sunil Chaudhari) #6

Should I share only un-commented configurations?

(Sunil Chaudhari) #7
        - C:\var\log\test.log
input_type: log
 exclude_lines: ["^DBG"]
 include_lines: ["^INFO"]
registry_file: "C:/ProgramData/filebeat/registry"

    hosts: ["localhost:9200"]


    to_files: true

  # To enable logging to files, to_files option has to be set to true

    path: C:\var\log\

    name: mybeat

    rotateeverybytes: 10485760 # = 10MB

  level: debug

(Tudor Golubenco) #8

Perhaps nothing is matching the include pattern. As it is now, it only catches lines starting with INFO. Maybe post a sample of the test.log as well.

(Sunil Chaudhari) #9
Apr 12 13:27:36 hostname SYSLOG 7196 - - INFO;2016-04-12T16:27:36.612+0300;CM;-;Generating all blacklist items as a binary response;-;-;-;-;blacklistRest.js;-;-
Apr 12 13:27:36 hostname SYSLOG 7186 - - INFO;2016-04-12T16:27:36.738+0300;CM;-;No blacklist items at all;-;-;-;-;blacklistRest.js;-;-

(Sunil Chaudhari) #10

Any guess what might be the issue?


(Tudor Golubenco) #11

Those lines don't start with INFO. Try changing the include_lines to contain just "INFO", not "^INFO".

(Steffen Siering) #12

try pattern '- INFO;'

(Sunil Chaudhari) #13

It was my mistake. file name was wrong. Actually windows was not showing the file extension. and hence mistake happened.

Anyways, It started and sending logs ot ES.
Now Ihave one question:
When I stop fb and start it again, it start harvester form the same offset position. Its good feature but , I want to carry out some tests on same lines of logs again and again. What should be done to reset harvestor position each time it starts?


(ruflin) #14

To reset the state, you have to remove the registry file which in your case I think is under "C:/ProgramData/filebeat/registry"

(Sunil Chaudhari) #15

Thank you all!

(system) #16