FileBeat not forwarding My IIS logs to elastic search


(fayaz) #1

Hi Guys

I am very new to Elastic stack. I am trying to setup a dashboard to monitor IIS logs. But the file beats is not working as expected. Here is my configuration.

Installed Version:6.6.1 Kibana, Elastic Search, Filebeat with IIs module enabled on my local machine.

The only different thing am doing here is I copied logs from my production server to my local machines and pointed file beat to that directory. But i am not seeing data on Kibana or elastic search.

Please suggest me what Is wrong in my configuration file.

Here is my configuration for filebeat.yml.

filebeat.inputs:

    - type: log

      # Change to true to enable this input configuration.
      enabled: false

      # Paths that should be crawled and fetched. Glob based paths.
      paths:
        # - E:/Share/W3SVC2/*.log
        #- /var/log/*.log
        #- c:\programdata\elasticsearch\logs\*

    # matching any regular expression from the list.
      exclude_lines: ['^#']
#============================= Filebeat modules ===============================
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: "C:/Program Files/filebeat/modules.d/*.yml"

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  reload.period: 10s

#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false
#============================== Kibana =====================================
setup.kibana:

  # Kibana Host
  host: "localhost:5601"

  # Kibana Space ID
  #space.id:
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Enabled ilm (beta) to use index lifecycle management instead daily indices.
  #ilm.enabled: false
# Configure processors to enhance or manipulate events generated by the beat.
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

IIS module configuration

- module: iis
  # Access logs
  access:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["E:/Share/W3SVC2/*.log"]

  # Error logs
  error:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

(fayaz) #2

My Filebeat logs are:

2019-03-05T10:06:02.873-0800	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 0
2019-03-05T10:06:02.873-0800	INFO	cfgfile/reload.go:150	Config reloader started
2019-03-05T10:06:33.235-0800	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":6515,"time":{"ms":6515}},"total":{"ticks":9968,"time":{"ms":9968},"value":9968},"user":{"ticks":3453,"time":{"ms":3453}}},"handles":{"open":223},"info":{"ephemeral_id":"ed48a92a-4dfd-4c2c-8e9a-1f25cc867fe4","uptime":{"ms":33596}},"memstats":{"gc_next":6174832,"memory_alloc":3788832,"memory_total":171348392,"rss":27144192}},"filebeat":{"events":{"active":5,"added":457,"done":452},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":1,"filtered":446,"total":447}}},"registrar":{"states":{"current":863,"update":453},"writes":{"success":452,"total":453}},"system":{"cpu":{"cores":8}}}}}
2019-03-05T10:06:47.263-0800	INFO	log/input.go:138	Configured paths: [E:\Share\W3SVC2\*.log]
2019-03-05T10:06:47.263-0800	INFO	elasticsearch/client.go:165	Elasticsearch url: http://localhost:9200
2019-03-05T10:06:47.270-0800	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.6.1
2019-03-05T10:06:47.273-0800	INFO	input/input.go:114	Starting input of type: log; ID: 13270352752995421963 
2019-03-05T10:07:02.868-0800	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10687,"time":{"ms":4172}},"total":{"ticks":16827,"time":{"ms":6859},"value":16827},"user":{"ticks":6140,"time":{"ms":2687}}},"handles":{"open":234},"info":{"ephemeral_id":"ed48a92a-4dfd-4c2c-8e9a-1f25cc867fe4","uptime":{"ms":63227}},"memstats":{"gc_next":5837536,"memory_alloc":5169240,"memory_total":321312440,"rss":2547712}},"filebeat":{"events":{"active":-5,"added":401,"done":406},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"pipeline":{"clients":1,"events":{"active":0,"filtered":412,"total":411}}},"registrar":{"states":{"current":863,"update":405},"writes":{"success":406,"total":405}}}}}
2019-03-05T10:07:32.868-0800	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":11375,"time":{"ms":688}},"total":{"ticks":17750,"time":{"ms":923},"value":17750},"user":{"ticks":6375,"time":{"ms":235}}},"handles":{"open":236},"info":{"ephemeral_id":"ed48a92a-4dfd-4c2c-8e9a-1f25cc867fe4","uptime":{"ms":93226}},"memstats":{"gc_next":5833424,"memory_alloc":3626888,"memory_total":326406744,"rss":-28672}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":863}}}}}
2019-03-05T10:08:02.869-0800	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":12171,"time":{"ms":796}},"total":{"ticks":18624,"time":{"ms":874},"value":18624},"user":{"ticks":6453,"time":{"ms":78}}},"handles":{"open":238},"info":{"ephemeral_id":"ed48a92a-4dfd-4c2c-8e9a-1f25cc867fe4","uptime":{"ms":123227}},"memstats":{"gc_next":5507296,"memory_alloc":4285872,"memory_total":331489752,"rss":32768}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":863}}}}}

(Pier-Hugues Pellerin) #3

@syedsfayaz In the provided log I do not see that the IIS module is enabled, can you add the listing of the files from C:/Program Files/filebeat/modules.d/*.yml

don't mind the above I see 2019-03-05T10:06:47.263-0800 INFO log/input.go:138 Configured paths: [E:\Share\W3SVC2\*.log] in the log which would mean the module is correctly configured.


(Pier-Hugues Pellerin) #4

It is the correct path? Because looking at the monitoring report there are currently no files open:

"harvester":{"open_files":0,"running":0}}

(fayaz) #5

Yes, It is the correct path. I have changed the logs directory to a different directory and I am able to see the logs now.

This is the logformat

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2019-02-26 00:00:01 **.**.**.*** GET /information/update.html - 443 - **.**.**.*** Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/72.0.3626.119+Safari/537.36 https://example.com/information/update.html 304 0 0 46

This is what I am seeing in kibana now. Do I need to add any other configuration for the logs to be parsed accordingly to my need.


(fayaz) #6

Provided Grok expressions do not match field value: [2018-12-31 12:52:33 10.44.0.136 GET /${#context['xwork.MethodAccessor.denyMethodExecution']=!(#_memberAccess['allowStaticMethodAccess']=true),(@java.lang.Runtime@getRuntime()).exec('ipconfig').waitFor()}.action - 443 - 10.50.6.188 Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+5.1;+Trident/4.0) - 404 0 2 0]

Any idea why its showing this error.


(Pier-Hugues Pellerin) #7

@syedsfayaz I did a quick look at the ingest pipeline that the ISS module use, I see that we use the URIPATH to parse the request path, so If we look at the pattern in ES I think it doesn't support a few characters used in the path.

So this looks like a bug to me on our side, can you create an issue on our tracker https://github.com/elastic/beats/ ?


(system) closed #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.