New Install ELK 6.1.0 & Filebeat 6.1.0

Elastic Stack Beats
1

I have just created a new install of ELK 6.1.0 and am trying to use Filebeat to ship IIS logs to logstash. The ELK stack appears to be working and the filebeat service has started. I've vreated firrewall exceptions on my ELK server for ports 9200, 9300, 5601, 5.44. Below are my settings:

logstash.yml:

 path.data: D:\Shares\ELK\Logstash\data
 path.settings: D:\Shares\ELK\Logstash\config
 path.logs: D:\Shares\ELK\Logstash\logs

iis_filter.conf:

input {
  beats {
    port => 5044
    type => "iis_log"
  }
}
filter {
  if [type] == "iis_log" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} (?:-|\"%{URIPATH:CS-URI-Query}\") %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
    }
  }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "filebeat_IIS-%{+YYYY.MM.dd}"
  }
}

filebeat.yml:

filebeat.prospectors:
- type: log
  enabled: false
  paths:
    - d:\Inetpub\logs\logfiles\*\*
  exclude_lines: ['#']
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.logstash:
  hosts: ["scamander.hstv.local:5044"]

After browsing to a couple of outr internal web sites I've gone into Kibana and I cannot create any Index patterns as it "Couldn't find any Elasticsearch data".

Is there something wrong with my configuration?

2

can you share your filebeat log file?

3

Sure. Again sorry for the delay.

2017-12-22T09:34:04Z WARN Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2017-12-26T16:51:57Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:65193->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-26T16:51:58Z ERR  Failed to publish events: write tcp 10.20.0.19:65193->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T08:49:05Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:65195->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T08:49:06Z ERR  Failed to publish events: write tcp 10.20.0.19:65195->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:22:58Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:65448->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:22:59Z ERR  Failed to publish events: write tcp 10.20.0.19:65448->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:29:59Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:65464->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:30:00Z ERR  Failed to publish events: write tcp 10.20.0.19:65464->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:31:04Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:65471->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:31:05Z ERR  Failed to publish events: write tcp 10.20.0.19:65471->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:33:04Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:65486->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:33:05Z ERR  Failed to publish events: write tcp 10.20.0.19:65486->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:36:01Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:65489->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:36:02Z ERR  Failed to publish events: write tcp 10.20.0.19:65489->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:44:03Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:65507->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:44:04Z ERR  Failed to publish events: write tcp 10.20.0.19:65507->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:45:58Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:65514->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T09:45:59Z ERR  Failed to publish events: write tcp 10.20.0.19:65514->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T11:33:58Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:65518->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T11:33:59Z ERR  Failed to publish events: write tcp 10.20.0.19:65518->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T11:35:03Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:49168->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T11:35:04Z ERR  Failed to publish events: write tcp 10.20.0.19:49168->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T11:53:02Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:49169->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T11:53:03Z ERR  Failed to publish events: write tcp 10.20.0.19:49169->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T11:55:02Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:49174->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T11:55:03Z ERR  Failed to publish events: write tcp 10.20.0.19:49174->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T11:57:03Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:49202->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T11:57:04Z ERR  Failed to publish events: write tcp 10.20.0.19:49202->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T12:04:05Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:49216->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T12:04:06Z ERR  Failed to publish events: write tcp 10.20.0.19:49216->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T12:11:06Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:49219->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T12:11:07Z ERR  Failed to publish events: write tcp 10.20.0.19:49219->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T12:14:01Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:49230->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T12:14:02Z ERR  Failed to publish events: write tcp 10.20.0.19:49230->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T13:15:01Z ERR  Failed to publish events caused by: write tcp 10.20.0.19:49238->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.
2017-12-27T13:15:02Z ERR  Failed to publish events: write tcp 10.20.0.19:49238->10.20.0.44:5044: wsasend: An existing connection was forcibly closed by the remote host.

Tony

4

Are these two topics related to the same issue? Missing field accessing 'path' accessing 'filebeat' (source:'filebeat.yml')

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.