Grok in Logstash not working


(Bikash Behuria) #1

I wanted to parse application logs which has information like timestamp , host name , message , log level etc. When I am using Grok pattern , I dont see the correct value is being assigned to the respscive variable.

For to make it more simple , I only tried to parse one string called Hostname like below

if[topics] == "ApplicationError" {
grok {
match => ["message", "%{WORD:hostserver}"]

And getting result in Message field not in hostserver variable. I checked with IPORHOST pattern also ..same result.

I checked in Grok debugger ..there also its saying no match. . Not sure what is the wrong in my pattern...

I am using Logstash 6.0.0

Below are my output


(Magnus Bäck) #2

Since the topics field isn't equal to "ApplicationError" the grok filter in question isn't run at all. Show the rest of your configuration.


(Bikash Behuria) #3

Hi , I have declared Topics and Type in input section

kafka {
bootstrap_servers => "localhost:9092"
topics => "ApplicationError"
type => "applicationerror"
}

if[topics] == "ApplicationError" {
grok {
match => ["message", "%{WORD:hostserver} %{WORD:severity} %{WORD:errormessage} %{WORD:appdomain} %{WORD:hostprocess} %{WORD:source}" ]

}
i also tried with Type field like

if[type] == "applicationerror" {
grok {
match => ["message", "%{WORD:hostserver} %{WORD:severity} %{WORD:errormessage} %{WORD:appdomain} %{WORD:hostprocess} %{WORD:source}" ]

}

I am doing some basic mistake i guess and even if I am trying with single word in input logs , its not working either in my config or not matching in Grok debugger.

I am very new to Logstash so need help

Thanks-Bikash


(Magnus Bäck) #4

I have declared Topics and Type in input section

I'm quite sure the value from the topics configuration option won't be copied to each event.

Anyway, what does an example message pulled from Kafka look like? If you use a stdout { codec => rubydebug } output, what do you get?


(Bikash Behuria) #5

Hi ,

This time , I removed the if condition so there is no check of topics or type before GROK parsing , here are the debug output


(Magnus Bäck) #6

If you want help, show your full configuration and copy/paste the text of an input message. Don't post screenshots.


(Bikash Behuria) #7

Hi , Now I am able to parse the message but still need help on different pattern I should use for Space , Path etc. Can you please help. Below are the details of Input output and configuration.

Input Logs
“SEGOTW10271196 Error This is a test message VTOMException This is a unit test Exception IN at Volvo.VTOM.UtilityComponentTests.Logging.VTOMLoggerTests.LogError() in C:\CAQv2\Source\VTOM\Tests\UnitTests\NVSComponents\Utilities\UtilityComponentTests\Logging\VTOMLoggerTests.cs:line 48 UnitTestAdapter: Running test C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\ENTERPRISE\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW\vstest.executionengine.x86.exe Volvo.VTOM.UtilityComponentTests.Logging.VTOMLoggerTests:Void LogError()”

Expected Output after GROK Parsing

Host Name - SEGOTW10271196
Severity – Error
Message - This is a test message VTOMException This is a unit test Exception IN at Volvo.VTOM.UtilityComponentTests.Logging.VTOMLoggerTests.LogError() in C:\CAQv2\Source\VTOM\Tests\UnitTests\NVSComponents\Utilities\UtilityComponentTests\Logging\VTOMLoggerTests.cs:line 48
appdomain - UnitTestAdapter: Running test
processname - C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\ENTERPRISE\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW\vstest.executionengine.x86.exe
Title - Volvo.VTOM.UtilityComponentTests.Logging.VTOMLoggerTests:Void LogError()

Actual Output

{
"severity" => "Error",
"errormessage" => "This",
"@timestamp" => 2018-01-02T09:52:52.732Z,
"hostserver" => "SEGOTW10271196",
"processname" => "a",
"@version" => "1",
"appdomain" => "is",
"message" => "SEGOTW10271196 Error This is a test message VTOMException This is a unit test Exception IN at Volvo.VTOM.UtilityComponentTests.Logging.VTOMLoggerTests.LogError() in C:\CAQv2\Source\VTOM\Tests\UnitTests\NVSComponents\Utilities\UtilityComponentTests\Logging\VTOMLoggerTests.cs:line 48 UnitTestAdapter: Running test C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\ENTERPRISE\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW\vstest.executionengine.x86.exe Volvo.VTOM.UtilityComponentTests.Logging.VTOMLoggerTests:Void LogError()",
"type" => "applicationerror",
"title" => "test"
}

My Conf is as below

input {
kafka {
bootstrap_servers => "localhost:9092"
topics => "ApplicationError"
type => "applicationerror"
}
}
filter {
grok { match => ["message", "%{WORD:hostserver} %{WORD:severity} %{WORD:errormessage} %{WORD:appdomain} %{WORD:processname} %{WORD:title}"] }

}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{type}"
}
stdout { codec => rubydebug }
}


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.