Grok Filter not extracting the fields

Elastic Stack Logstash
1

Hey, My grok filter is working but not extracting and here is the logstash.conf

    input { 
      beats {
        port => 5044
      }
    }
    filter {
      if [type] == "iis" {
    		if [message] =~ "^#" {
                    drop {}
            }
    	   grok {
    			match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} (?:-|\"%{URIPATH:CS-URI-Query}\") %{NUMBER:Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
    		}
        }
    }	
    output {
       elasticsearch {
        hosts => ["locahost:5555"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
      }
    }

and here is filebeat.config

filebeat.prospectors:
    - input_type: log
      paths:
        - c:\inetpub\logs\LogFiles\W3SVC1\*.log
      document_type: iis

    output.logstash:
      hosts: ["localhost"]

and kibana output:

@timestamp
July 12th 2017, 09:13:50.545
t @version
1
t _id

t _index
filebeat-2017.07.12

_score

 -

t _type
log
t beat.hostname
symwork
t beat.name
symwork
t beat.version
5.4.0
t host
symwork
t input_type
log
t message
2017-07-12 14:12:59 W3SVC1 symworkflow ip POST /Green/WorkflowManagementService.asmx - 900- clientip HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+) - - b.p.com 500 1 1 900 700 9

offset

1,229,523

t source
c:\inetpub\logs\LogFiles\W3SVC1\u_ex170712.log
t tags
beats_input_codec_plain_applied
t type
log

2

The type field contains "log" but you're only applying the grok filter to "iis" events.

If you format your Filebeat configuration as preformatted text it won't be damaged when you post it and we can probably point out what's wrong.

3

Thanks for reply. I'm new to logstash and elastic. I just preformatted the text for filebeat

4

Hmm, looks correct. I'm not sure what's up here.

5

Thats What I thought. I restarted the logstash container and filebeat in windowsserver and kibana. I'm not sure why is not extracting the field

6

Figure It out. I was missing - type => iis in the logstash input field

7

That works too, but makes it harder to collect different kinds of logs with the same beats input.

8

That makes sense. If I don't put in the input type, It wont work.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.