Extract timestamp from my log file along with loglevel?

Elastic Stack Logstash
1

I am using grok to extract the logtime alongwith loglevel but everytime i got grokparsefailure error. Although on individual selection they work fine.

Here are the details:

My log file:

2018-01-31 07:35:49.899 [Information] Request starting HTTP/1.1 POST http://ezcustomers-dev.ahcs.com/API/SiteUsers/GetAvailableSitesForUser application/json; charset=utf-8 92
2018-01-31 07:35:50.592 [Information] Executing action method "ezCustomerAPI.Controllers.SiteUserController.GetAvailableSitesForUser (ezCustomerAPI)" with arguments (["Models.SiteUsers.UserSiteByAppDTO"]) - ModelState is Valid

Logstash.conf:

input {
beats{
port=> "5044"
}
}

filter {
grok{
match =>{"message"=>"%{TIMESTAMP_ISO8601:logtime}%{LOGLEVEL:Loglevel}"}
}
}

output {
stdout { codec => rubydebug }
}

2

I am using grok to extract the logtime alongwith loglevel but everytime i got grokparsefailure error.

Yes, because

  • you're not taking the square brackets into account (it's "[Information]" not "Information"), and
  • you don't have a space between %{TIMESTAMP_ISO8601:logtime} and %{LOGLEVEL:Loglevel}.

Although on individual selection they work fine.

What do you mean?

3

I mean When I only give grok this:

filter {
grok{
match =>{"message"=>"%{LOGLEVEL:Loglevel}"}
}
}

It works fine and also for timestamp.

4

However, I resolved it with this:

filter {
grok{
match =>{"message"=>"%{LOGLEVEL:Loglevel}"}
}
grok{
match =>{"message"=>"%{TIMESTAMP_ISO8601:logtime}"}
}
}

But don't know if this is the best way to do this or not.

5

No, that's not the best way. Your original idea is fine but there are a few bugs in the expression, as I pointed out.

6

When I use:

%{TIMESTAMP_ISO8601:logtime} \[%{WORD:LogLevel}\]

The result is this:

{
  "logtime": [
    [
      "2018-01-31 07:35:49.899"
    ]
  ],
  "LogLevel": [
    [
      "Information"
    ]
  ]
}

If I use this:

%{TIMESTAMP_ISO8601:logtime} \[%{LOGLEVEL:LogLevel}

I get this:

{
  "logtime": [
    [
      "2018-01-31 07:35:49.899"
    ]
  ],
  "LogLevel": [
    [
      "Info"
    ]
  ]
}

The %{LOGLEVEL:LogLevel} is translated into Info, now I do not know if you would expect the complete word "Information" but it is something to be aware of. Also as you can see, in the grok the last \] is not there. If you add it the grok filter will not match. Now it is not clear if you need more fields grok out of those log lines but is might pose a problem in the future.

I hope this helps.