Parse errors for IIS logs using ELK

Elastic Stack Logstash
1

My Logs are like below:

2017-06-14T10:39:30.105Z x.x.x.x 2017-06-14 10:39:27,827 Thread:'9' Level:'ERROR' Message:System error: Get security settings list service call failed. [[some.app]]some.app.ServiceResponseWrapperException: Get security settings list service call failed.^M
   at some.app.Utilities.SecuritySettingsHelper.get_Settings()^M
   at some.app.Utilities.SecuritySettingsHelper.LookupSetting(String key)^M
   at some.app.ApplicationUserManager.Create(IdentityFactoryOptions`1 options, IOwinContext context)^M
   at Microsoft.AspNet.Identity.Owin.IdentityFactoryMiddleware`2.<Invoke>d__0.MoveNext()^M
--- End of stack trace from previous location where exception was thrown ---^M
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)^M
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)^M
   at some.app.Middleware.RequestTracingMiddleware.<Invoke>d__4.MoveNext()^M
--- End of stack trace from previous location where exception was thrown ---^M
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)^M
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)^M
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()^M
--- End of stack trace from previous location where exception was thrown ---^M
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)^M
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)^M
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.<DoFinalWork>d__2.MoveNext()^M
--- End of stack trace from previous location where exception was thrown ---^M
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar)^M
   at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()^M
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)^M
   
2017-06-14T11:00:04.619Z x.x.x.x 2017-06-14 11:00:00,001 Thread:'DefaultQuartzScheduler_Worker-5' Level:'INFO' Message:Starting scheduled job Workflow.
2017-06-14T11:00:05.739Z x.x.x.x 2017-06-14 11:00:00,001 Thread:'DefaultQuartzScheduler_Worker-5' Level:'INFO' Message:Starting scheduled job Workflow.
2017-06-14T11:00:08.439Z x.x.x.x 2017-06-14 11:00:00,001 Thread:'DefaultQuartzScheduler_Worker-5' Level:'INFO' Message:Starting scheduled job Workflow.
2017-06-14T11:00:04.623Z x.x.x.x 2017-06-14 11:00:00,005 Thread:'DefaultQuartzScheduler_Worker-5' Level:'INFO' Message:Finished scheduled job Workflow.
2017-06-14T11:00:13.921Z x.x.x.x 2017-06-14 11:00:00,001 Thread:'DefaultQuartzScheduler_Worker-5' Level:'INFO' Message:Starting scheduled job Workflow.
2017-06-14T11:00:05.749Z x.x.x.x 2017-06-14 11:00:00,005 Thread:'DefaultQuartzScheduler_Worker-5' Level:'INFO' Message:Finished scheduled job Workflow.
2017-06-14T11:00:08.443Z x.x.x.x 2017-06-14 11:00:00,005 Thread:'DefaultQuartzScheduler_Worker-5' Level:'INFO' Message:Finished scheduled job Workflow.
2017-06-14T11:00:13.924Z x.x.x.x 2017-06-14 11:00:00,004 Thread:'DefaultQuartzScheduler_Worker-5' Level:'INFO' Message:Finished scheduled job Workflow.

GROK filter is as below:

filter {
    multiline {
        pattern => "^%{TIMESTAMP_ISO8601}"
        negate => true
        what => previous
    }

grok {
    match => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{GREEDYDATA:information}"]
    tag_on_failure => ["error_message_not_parsed"]
    remove_field => ["message"]
    break_on_match => false
}

grok {
    match => [ "path", "/Logs/(?<server>[^/]+)/(.*).*" ]
    tag_on_failure => ["path_not_parsed"]
}

}

And the appender is as below:

<appender name="log4net" type="log4net.Appender.UdpAppender">
    <param name="RemoteAddress" value="x.x.x.x" />
      <param name="RemotePort" value="8082" />
      <layout type="log4net.Layout.PatternLayout" value="%utcdate{ISO8601} Thread:'%thread' Level:'%level' Message:%message" />
  </appender>

These are the fields in Kibana

Selected Fields

Available Fields 
message
@timestamp
@version
_id
_index
_score
_type
host
tags
type

What I really need is the fields like error level, thread, http response, etc to be parsed and shown as additional fields to select and analyse if needed.

I am also getting tags:error_message_not_parsed, path_not_parsed in kibana so, parsing and filtering is not working.

Please advise.

2

Your grok expression

(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{GREEDYDATA:information}

clearly doesn't match what a log entry looks like:

2017-06-14T10:39:30.105Z x.x.x.x 2017-06-14 10:39:27,827 Thread:'9' Level:'ERROR' Message:System error: Get security settings ...
3

The above log is from the logs received by logstash and saved at /var/log/logstash/output_line.log

I also tried : match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:hostname} %{TIMESTAMP_ISO8601:log_time} %{DATA:thread} %{LOGLEVEL:log-level} %{GREEDYDATA:information}"]

but that doesnt parse anything. Could you tell me whats wrong or a very basic filter which will list thread, loglevel, timestamp and message ?

4

The above log is from the logs received by logstash and saved at /var/log/logstash/output_line.log

Yes, I suspected that but it wasn't very clear from your question.

Something like this should work:

%{TIMESTAMP_ISO8601} Thread:'(?<thread>[^']+)' Level:'%{LOGLEVEL:log-level}' Message:%{GREEDYDATA:information}
5

Thank you. This worked !

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.