IIS logs showing as a string

[Venu1]

Hello all, I'm a newbie to ELK,
I'm able to send the IIS logs and other application logs to the Elastic Kibana from the servers directly without logstash, IIS logs are showing up one string in the Elastic Kibana, I would like them to appear as individual fields out

In order to achieve this is logstash mandatory?
or can I be able to edit the IIS pipeline.yml or any other config file to achieve this? can someone provide a sample config file that needs to be modified?

Elasticsearch version: v 7.10.2
Filebeat version: filebeat-7.10.2-windows-x86_64

[stephenb (Stephen Brown)]

Hi @Venu1 Welcome to the community!

1st please do not post screen shots of text, they are hard to read, can not be searched, some folks can not even view them please post them as text and use the </> to format them.

No logstash is not neccessary.

Did you enable the IIS Module in filbeat? See here It is purpose built for this.

If so are you saying you have properly configured and enabled the IIS module and that it is the log line is still not parsed? Perhaps you have enabled custom fields in the IIS logs.

Yes you can make a copy and edit update the filebeat IIS ingest pipeline if needed... and then call / use that.

But of course since we don't have a sample of your log in text we would not be able to create one for you... simple anonymize the fields.

[Venu1]

@stephenb,
thank you for confirming that I don't need Logstash

IIS logs are coming to the dashboard but the problem is in a single string, we want them to break into fields so each field is searchable, in order to achieve this what changes need to be done? I think grok in pipeline.yml need to be changed, looking at our sample IIS log below can you please provide us pipeline.yml file or any other file that need to be changed

please see below fields and sample IIS logs

#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-user.name c-ip cs-version cs(User- agent) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2021-10-24 22:00:00 W3SVC1 XXCIQWBPD03 151.xxx.154.79 POST /xxdotnet/Login/LoginInfo.asmx/GetExpenseCode - 8080 - 151.xxx.154.79 HTTP/1.1 Mozilla/4.0+(compatible;+Win32;+WinHttp.WinHttpRequest.5) - 151.xxx.154.79:8080 200 0 0 452 428 10

yes, IIS module is defined looks like below

# Module: iis
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-iis.html

- module: iis
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["D:/*/*/*/W3SVC*/u_*","C:/*/*/*/W3SVC*/u_*"]

  # Error logs
  error:
    enabled: true
    var.paths: ["D:/Logs/HTTPERR/*.log"]
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

Thank you,
Venu

[stephenb (Stephen Brown)]

@Venu1

2021-10-24 22:00:00 W3SVC1 XXCIQWBPD03 151.xxx.154.79 POST /xxdotnet/Login/LoginInfo.asmx/GetExpenseCode - 8080 - 151.xxx.154.79 HTTP/1.1 Mozilla/4.0+(compatible;+Win32;+WinHttp.WinHttpRequest.5) - 151.xxx.154.79:8080 200 0 0 452 428 10

So I created a new grok... you should double check it... you can add this through the UI ( I think that UI was in 7.10.. you should upgrade!) ... or through the API. You can add it to the existing pipeline or you can create your own by copying it to a new name... if you do that see the bottom.

You should learn how to do it... I just lined up the definition with the other grok pattern and rearranged..

#Fields: date time                    s-sitename                            s-computername                            s-ip                             cs-method                        cs-uri-stem                     cs-uri-query                     s-port                                 cs-user.name               c-ip                             cs-version                         cs(User- agent)                      cs(Referer)                             cs-host                              sc-status                                      sc-substatus                                sc-win32-status                              sc-bytes                                      cs-bytes                                    time-taken
%{TIMESTAMP_ISO8601:iis.access.time} (?:-|%{NOTSPACE:iis.access.site_name}) (?:-|%{NOTSPACE:iis.access.server_name}) (?:-|%{IPORHOST:source.address}) (?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:_tmp.url_path}) (?:-|%{NOTSPACE:_tmp.url_query}) (?:-|%{NUMBER:destination.port:long}) (?:-|%{NOTSPACE:user.name}) (?:-|%{IPORHOST:source.address}) (?:-|HTTP/%{NUMBER:http.version}) (?:-|%{NOTSPACE:user_agent.original}) (?:-|%{NOTSPACE:http.request.referrer}) (?:-|%{NOTSPACE:destination.domain}) (?:-|%{NUMBER:http.response.status_code:long}) (?:-|%{NUMBER:iis.access.sub_status:long}) (?:-|%{NUMBER:iis.access.win32_status:long}) (?:-|%{NUMBER:http.response.body.bytes:long}) (?:-|%{NUMBER:http.request.body.bytes:long}) (?:-|%{NUMBER:temp.duration:long})

After I added it as the top grok pattern

POST /_ingest/pipeline/filebeat-7.14.1-iis-access-pipeline/_simulate
{
  "docs": [
    {
      "_index": "m-index",
      "_id": "kMpUTHoBr7SFhhL5-98P",
      "_source": {
        "@timestamp" : "2021-11-09T03:17:20.0516241Z",
        "message": "2021-10-24 22:00:00 W3SVC1 XXCIQWBPD03 151.xxx.154.79 POST /xxdotnet/Login/LoginInfo.asmx/GetExpenseCode - 8080 - 151.001.154.79 HTTP/1.1 Mozilla/4.0+(compatible;+Win32;+WinHttp.WinHttpRequest.5) - 151.010.154.79:8080 200 0 0 452 428 10"      }
    }
  ]
}

Results

{
  "docs" : [
    {
      "doc" : {
        "_index" : "m-index",
        "_type" : "_doc",
        "_id" : "kMpUTHoBr7SFhhL5-98P",
        "_source" : {
          "temp" : { },
          "destination" : {
            "port" : 8080,
            "domain" : "151.010.154.79:8080"
          },
          "source" : {
            "address" : "151.xxx.154.79",
            "ip" : "151.xxx.154.79"
          },
          "error" : {
            "message" : "'151.xxx.154.79' is not an IP string literal."
          },
          "url" : {
            "path" : "/xxdotnet/Login/LoginInfo.asmx/GetExpenseCode",
            "extension" : "asmx/GetExpenseCode",
            "original" : "/xxdotnet/Login/LoginInfo.asmx/GetExpenseCode",
            "scheme" : null,
            "domain" : "151.010.154.79:8080"
          },
          "iis" : {
            "access" : {
              "site_name" : "W3SVC1",
              "server_name" : "XXCIQWBPD03",
              "sub_status" : 0,
              "win32_status" : 0
            }
          },
          "@timestamp" : "2021-10-24T22:00:00.000Z",
          "http" : {
            "request" : {
              "method" : "POST",
              "body" : {
                "bytes" : 428
              }
            },
            "version" : "1.1",
            "response" : {
              "body" : {
                "bytes" : 452
              },
              "status_code" : 200
            }
          },
          "event" : {
            "duration" : 10000000,
            "ingested" : "2021-11-09T03:20:38.642371800Z",
            "original" : "2021-10-24 22:00:00 W3SVC1 XXCIQWBPD03 151.xxx.154.79 POST /xxdotnet/Login/LoginInfo.asmx/GetExpenseCode - 8080 - 151.001.154.79 HTTP/1.1 Mozilla/4.0+(compatible;+Win32;+WinHttp.WinHttpRequest.5) - 151.010.154.79:8080 200 0 0 452 428 10",
            "created" : "2021-11-09T03:17:20.0516241Z"
          },
          "user_agent" : {
            "name" : "Other",
            "original" : "Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)",
            "os" : {
              "name" : "Windows",
              "version" : "95",
              "full" : "Windows 95"
            },
            "device" : {
              "name" : "Spider"
            }
          }
        },
        "_ingest" : {
          "timestamp" : "2021-11-09T03:20:38.6423718Z"
        }
      }
    }
  ]
}

If you change the pipeline name you will need to overwrite it.. in the iis.yml file NOT at the output.elasticsearch section in the filebeat.yml file ... as the input overrides that and thus the default pipeline will be used.

As shown

  # Access logs
  access:
    enabled: true
    input:
      pipeline: my-new-custom-pipeline

Pick of Ingest Pipeline UI, put the new line in and drag it to the top.