Filebeat seems not interact with Logstash - Windows

Elastic Stack Beats
1

Hi, Colleagues!

I have installed ELK in Windows and runs successfully, creatings all indexes.
But it need to be integrate with Filebeat. Then, the structure is FB > LS > ES.
But, despite of promt says "2016/01/08 11:08:13.170670 outputs.go:111: INFO Activated logstash as output plugin.", the indices have not been created in Elasticsearch.

Steps:
Iniciate Elasticsearch - OK

Iniciate Logstash - OK
cd C:\logstash-2.1.1\bin logstash -f logstash.conf

Iniciate Filebeat - OK
cd C:\filebeat-1.0.1-windows .\filebeat.exe -e -c .\filebeat.yml -d "service"

When I check in Elasticsearch, the indeces have not been created.

Configuration

Logstash - logstash-2.1.1 - logstash.conf

input {
  beats {
    port => 5044
	type => "systemout_teste"
  }
}

filter {
	if [type] == "systemout_teste" {
              ...
}
}

output {
  elasticsearch {
	hosts => ["localhost:9200"] 
	index => "systemout_teste"
  }
}

Filebeat - filebeat-1.0.1-windows - filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - C:\files\SystemOut.log
      input_type: log

      document_type: systemout_teste

  registry_file: "C:\filebeat-1.0.1-windows\registry"
  config_dir:

output:
  logstash:
    hosts: ["localhost:5044"]
    worker: 1

Outputs

Logstash

io/console not supported; tty will not be manipulated
Settings: Default filter workers: 1
Logstash startup completed

Filebeat

2016/01/08 11:51:57.498070 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/01/08 11:51:57.504070 outputs.go:111: INFO Activated logstash as output plugin.
2016/01/08 11:51:57.504070 publish.go:249: INFO Publisher name: CTS08152138
2016/01/08 11:51:57.510070 beat.go:107: INFO Init Beat: filebeat; Version: 1.0.1
2016/01/08 11:51:57.510070 beat.go:133: INFO filebeat sucessfully setup. Start running.
2016/01/08 11:51:57.511070 registrar.go:66: INFO Registry file set to: C:\filebeegistry1-windows\♀ilebeat-1.0.1-windows
2016/01/08 11:51:57.512070 service_windows.go:49: DBG  Windows is interactive: true
2016/01/08 11:51:57.512070 crawler.go:78: INFO All prospectors initialised with0 states to persist
2016/01/08 11:51:57.512070 registrar.go:83: INFO Starting Registrar
2016/01/08 11:51:57.512070 spooler.go:77: INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016/01/08 11:51:57.512070 filebeat.go:122: INFO Start sending events to output

As I said above, seems like all is working, but no index was created. Anyone knows how to solve this situation?

Thanks a lot!

2

Hm... can you start filebeat with -d ?

3

The -d command which you are saying is " cd C:\filebeat-1.0.1-windows .\filebeat.exe -e -c .\filebeat.yml -d "service" "?

4

You have to run filebeat with .\filebeat.exe -e -c .\filebeat.yml -d '*'

5

I have runned with '', but the indices still not being created.
Below is the output drom Power Shell:
C:\filebeat-1.0.1-windows>.\filebeat.exe -e -c .\filebeat.yml -d ""
2016/01/11 16:17:09.307195 beat.go:97: DBG Initializing output plugins
2016/01/11 16:17:09.308195 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/01/11 16:17:09.321195 outputs.go:111: INFO Activated logstash as output plugin.
2016/01/11 16:17:09.321195 publish.go:198: DBG create output worker: 0x0, 0x0
2016/01/11 16:17:09.321195 publish.go:235: DBG No output is defined to store the topology. The server fields might not be filled.

2016/01/11 16:17:09.321195 publish.go:249: INFO Publisher name: CTS08152138
2016/01/11 16:17:09.324195 async.go:95: DBG create bulk processing worker (interval=1s, bulk size=200)
2016/01/11 16:17:09.325195 beat.go:107: INFO Init Beat: filebeat; Version: 1.0.1

2016/01/11 16:17:09.325195 beat.go:133: INFO filebeat sucessfully setup. Start running.
2016/01/11 16:17:09.326195 registrar.go:66: INFO Registry file set to: C:\filebeegistry1-windows\♀ilebeat-1.0.1-windows

2016/01/11 16:17:09.327195 spooler.go:44: DBG Set idleTimeoutDuration to 5s
2016/01/11 16:17:09.327195 crawler.go:38: DBG File Configs: [C:\files\SystemOut.log]

2016/01/11 16:17:09.328195 prospector.go:128: DBG Set ignore_older duration to 24h0m0s
2016/01/11 16:17:09.328195 prospector.go:128: DBG Set scan_frequency duration to 10s
2016/01/11 16:17:09.329195 prospector.go:128: DBG Set backoff duration to 1s
2016/01/11 16:17:09.330195 prospector.go:128: DBG Set max_backoff duration to 10s
2016/01/11 16:17:09.331195 prospector.go:128: DBG Set partial_line_waiting duration to 5s
2016/01/11 16:17:09.331195 crawler.go:58: DBG Waiting for 1 prospectors to initialise
2016/01/11 16:17:09.327195 service_windows.go:49: DBG Windows is interactive: true
2016/01/11 16:17:09.328195 spooler.go:77: INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016/01/11 16:17:09.332195 prospector.go:141: DBG Harvest path: C:\files\SystemOut.log
2016/01/11 16:17:09.333195 prospector.go:207: DBG scan path C:\files\SystemOut.log
2016/01/11 16:17:09.382195 prospector.go:219: DBG Check file for harvesting: C:\files\SystemOut.log

2016/01/11 16:17:09.382195 prospector.go:273: DBG Start harvesting unknown file: C:\files\SystemOut.log

2016/01/11 16:17:09.382195 prospector.go:289: DBG Fetching old state of file to resume: C:\files\SystemOut.log
2016/01/11 16:17:09.383195 prospector.go:306: DBG Skipping file (older than ign
ore older of 24h0m0s, 676h43m14.413036s): C:\files\SystemOut.log
2016/01/11 16:17:09.383195 prospector.go:207: DBG scan path C:\files\SystemOut.log
2016/01/11 16:17:09.384195 prospector.go:219: DBG Check file for harvesting: C:\files\SystemOut.log
2016/01/11 16:17:09.384195 prospector.go:341: DBG Update existing file for harvesting: C:\files\SystemOut.log
2016/01/11 16:17:09.385195 prospector.go:383: DBG Not harvesting, file didn't change: C:\files\SystemOut.log
...

Notice:
No registry file was created.

Is this right?

6

See debug output:

2016/01/11 16:17:09.383195 prospector.go:306: DBG Skipping file (older than ignore older of 24h0m0s, 676h43m14.413036s): C:\files\SystemOut.log
...
2016/01/11 16:17:09.385195 prospector.go:383: DBG Not harvesting, file didn't change: C:\files\SystemOut.log

You file seems to be very very old and ignore_older options ensures file is not collected. Update the file's last modified timestamp (e.g. https://support.microsoft.com/en-us/kb/69581).

7

This is the "error" that don't creates the indices. Problem solved.

Thanks a lot Steffens!

8

I am New to Elastic and I have a similar config. I'm using two sets of beats winlog beat and file beat. the winlog beat is fine and it created an index and I see data.... However file beat is not creating an index. I performed the test as mention above and this is what I get as an output.. Please help.

PS C:\filebeat> .\filebeat.exe -e -c .\filebeat.yml -d '*'
.\filebeat.exe : 2016/07/26 18:41:21.104877 beat.go:135: DBG Initializing output plugins
At line:1 char:2

  • .\filebeat.exe -e -c .\filebeat.yml -d '*'
  •  + CategoryInfo          : NotSpecified: (2016/07/26 18:4... output plugins:String) [], RemoteException
 + FullyQualifiedErrorId : NativeCommandError

2016/07/26 18:41:21.104877 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/07/26 18:41:21.104877 client.go:297: DBG ES Ping(url=http://192.168.110.100:9200, timeout=1m30s)
2016/07/26 18:41:21.106847 client.go:306: DBG Ping status code: 200
2016/07/26 18:41:21.106847 outputs.go:126: INFO Activated elasticsearch as output plugin.
2016/07/26 18:41:21.106847 file.go:39: INFO File output base filename set to: filebeat
2016/07/26 18:41:21.106847 file.go:50: INFO Rotate every bytes set to: 10485760
2016/07/26 18:41:21.106847 file.go:57: INFO Number of files set to: 7
2016/07/26 18:41:21.107349 outputs.go:126: INFO Activated file as output plugin.
2016/07/26 18:41:21.107349 publish.go:232: DBG Create output worker
2016/07/26 18:41:21.107349 publish.go:232: DBG Create output worker
2016/07/26 18:41:21.107349 publish.go:274: DBG No output is defined to store the topology. The server fields might not be filled.
2016/07/26 18:41:21.107349 publish.go:288: INFO Publisher name: ##$$ Computer name
2016/07/26 18:41:21.114908 async.go:78: INFO Flush Interval set to: 1s
2016/07/26 18:41:21.114908 async.go:84: INFO Max Bulk Size set to: 50
2016/07/26 18:41:21.114908 async.go:92: DBG create bulk processing worker (interval=1s, bulk size=50)
2016/07/26 18:41:21.114908 async.go:78: INFO Flush Interval set to: -1ms
2016/07/26 18:41:21.114908 async.go:84: INFO Max Bulk Size set to: -1
2016/07/26 18:41:21.114908 beat.go:147: INFO Init Beat: filebeat; Version: 1.2.3
2016/07/26 18:41:21.116912 beat.go:173: INFO filebeat sucessfully setup. Start running.
2016/07/26 18:41:21.116912 registrar.go:68: INFO Registry file set to: C:\ProgramData\filebeat\registry
2016/07/26 18:41:21.116912 registrar.go:80: INFO Loading registrar data from C:\ProgramData\filebeat\registry
2016/07/26 18:41:21.116912 spooler.go:44: DBG Set idleTimeoutDuration to 5s
2016/07/26 18:41:21.116912 crawler.go:38: DBG File Configs: [C:\Filebeat\log*.txt]
2016/07/26 18:41:21.116912 prospector.go:133: INFO Set ignore_older duration to 0
2016/07/26 18:41:21.116912 prospector.go:133: INFO Set close_older duration to 1h0m0s
2016/07/26 18:41:21.116912 prospector.go:133: INFO Set scan_frequency duration to 10s
2016/07/26 18:41:21.116912 prospector.go:93: INFO Input type set to: log
2016/07/26 18:41:21.116912 prospector.go:133: INFO Set backoff duration to 1s
2016/07/26 18:41:21.116912 prospector.go:133: INFO Set max_backoff duration to 10s
2016/07/26 18:41:21.116912 prospector.go:113: INFO force_close_file is disabled
2016/07/26 18:41:21.116912 crawler.go:58: DBG Waiting for 1 prospectors to initialise
2016/07/26 18:41:21.116912 prospector.go:143: INFO Starting prospector of type: log
2016/07/26 18:41:21.116912 prospector.go:161: DBG exclude_files: []
2016/07/26 18:41:21.116912 prospector.go:261: DBG scan path C:\Filebeat\log*.txt
2016/07/26 18:41:21.116912 prospector.go:261: DBG scan path C:\Filebeat\log*.txt
2016/07/26 18:41:21.116912 crawler.go:65: DBG No pending prospectors. Finishing setup
2016/07/26 18:41:21.116912 crawler.go:78: INFO All prospectors initialised with 0 states to persist
2016/07/26 18:41:21.116912 registrar.go:87: INFO Starting Registrar
2016/07/26 18:41:21.116912 publish.go:88: INFO Start sending events to output
2016/07/26 18:41:21.116912 service_windows.go:49: DBG Windows is interactive: true
2016/07/26 18:41:21.116912 spooler.go:77: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2016/07/26 18:41:23.617957 spooler.go:97: DBG Flushing spooler because of timeout. Events flushed: 0
2016/07/26 18:41:31.117402 prospector.go:185: DBG Start next scan

9

And this is my filebeat configuration file aggregated so it can fit

filebeat:

List of prospectors to fetch data.

prospectors:

    - C:\Filebeat\log\*.txt
   

  encoding: utf-8


  input_type: log
  document_type: log
  registry_file: "C:/ProgramData/filebeat/registry"
  utput:

Elasticsearch as output

elasticsearch:

hosts: ["192.168.110.100:9200"]
index: "filebeat"

File as output

file:
# Path to the directory where to save the generated files. The option is mandatory.
path: C:\filebeat\Logs\outPut

# Name of the generated files. The default is `filebeat` and it generates files: `filebeat`, `filebeat.1`, `filebeat.2`, etc.
filename: filebeat

logging:
rotateeverybytes: 10485760 # = 10MB
10

@alvaroca1 Can you please open a new topic with your question as the previous problem seems to be resolved.

11

@ruflin will do.

12

All done here is the link to the new topic for help FileBeat not sending info to elastic