Importing IIS logs and matching it with GROK and adding additional fields

[W.Almansoori]

Hello there.

I've been trying to import some logs from an IIS server. I configured the filebeat.yml, enabled inputs and set the path.
I've enabled the IIS module as well and configured the path of the access logs.

Starting filebeat does import the data but it only imports the very first line and skips the rest.

This is the filebeat.yml:

--- 
filebeat.config.modules: 
  path: "${path.config}/modules.d/*.yml"
  reload.enabled: false
filebeat.inputs: 
  - 
    enabled: true
    paths: 
      - "C:\\Users\\Elastic\\Documents\\*.log"
    type: log
output.elasticsearch: 
  hosts: 
    - "localhost:9200"
processors: 
  - 
    add_host_metadata: ~
  - 
    add_cloud_metadata: ~
  - 
    add_docker_metadata: ~
  - 
    add_kubernetes_metadata: ~
setup.kibana: ~
setup.template.settings: 
  index.number_of_shards: 1

#Fields: date time cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2019-11-18 08:48:20 GET /de adpar=12345&gclid=1234567890 443 - 149.172.138.41 HTTP/2.0 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/62.0.3202.89+Safari/537.36+OPR/49.0.2725.39 - https://www.google.de/ www.site-logfile-explorer.com 301 0 0 624 543 46
2018-11-18 08:48:20 GET /de/ adpar=12345&gclid=1234567890 443 - 149.172.138.41 HTTP/2.0 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/62.0.3202.89+Safari/537.36+OPR/49.0.2725.39 - https://www.google.de/ www.site-logfile-explorer.com 200 0 0 12973 544 62

Currently when I create an index with the data from the filebeat and then head to the Discover tab to view the filebeat, there is data. However the message section of that entry is just

#Fields: date time cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken

The data below that, as in the log file, isn't parsed or displayed.

Furthermore, after parsing the data I would like it to be labelled according to the GROK pattern, which is the same as the fields above and maybe try to add additional fields.

Any clue what could be wrong with the parsing process?

Best regards.

[MarianaD]

hi @W.Almansoori, can you clarify here? Are you using the IIS module (if so, can you provide us with the configuration?)
In the config above I see you are using a filebeat log input, are those logs unrelated to the iis logs?

[W.Almansoori]

Hi.

Yes, I am using the IIS module. Here are is the configuration

- module: iis
  # Access logs
  access:
    enabled: true
     var.paths: ["C:/Users/Elastic/Documents/*.log"]

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Error logs
  error:
    enabled: true

In the config, filebeat.yml, the path leads to where the IIS logs are stored.