Creating Grok Pattern for logstash config

sunny1 · August 25, 2017, 1:07am

Hello everyone out there!
I recently started using ELK so sorry if my question silly.

I need help with modifying my grok filter to achieve particular set of data. Tried lot but could not so looking for help.

Log file sample:--

2017-08-23 16:00:02,106 INFO : Test run configuration: RunId: a6e5a151-197a-48d7-86c5-6714ced296b7, Team: Learning, FrameWork:CTFXUnit, Branch: release, Environment:QA05, Portal:QA052_AUTO_LMS_SIGNOFF, Browser:Chrome, EnableDatabaseLog:True, EnableTestRailLog:True, TestRailConfiguration:Milestone: , TestRunType: TestPlan, TestPlan:LMS_Regression_Xunit, UpdateExistingTestRailRun: True., DebugLogLevel:2, TraitFilters:Name:Category, Operator:Equal, Value:Dothraki,
2017-08-23 16:00:02,121 INFO : Test Run:a6e5a151-197a-48d7-86c5-6714ced296b7, Status: Ready.
2017-08-23 16:00:05,377 INFO : Downloading items from TFS please wait...
2017-08-23 16:01:15,357 INFO : Runing batch file:'RestoreNuGetPackages.bat' started.
2017-08-23 16:01:18,693 INFO : Runing batch file:'RestoreNuGetPackages.bat' completed
2017-08-23 16:01:18,693 INFO : Test Run:a6e5a151-197a-48d7-86c5-6714ced296b7, Status: SourceControlFilesCopied.
2017-08-23 16:01:18,946 INFO : Runing batch file:'C:\temp\a6e5a151-197a-48d7-86c5-6714ced296b7\TestScript\LMS.XUnit.ObjectModels\AutoItX\RegisterAutoIt.bat' started.
2017-08-23 16:01:19,143 INFO : Runing batch file:'C:\temp\a6e5a151-197a-48d7-86c5-6714ced296b7\TestScript\LMS.XUnit.ObjectModels\AutoItX\RegisterAutoIt.bat' completed
2017-08-23 16:01:19,329 INFO : MSBuild in progress...
2017-08-23 16:01:39,354 INFO : MSBuild completed...
2017-08-23 16:01:39,354 INFO : Test Run:a6e5a151-197a-48d7-86c5-6714ced296b7, Status: MSBuildCompleted.
2017-08-23 16:01:39,594 INFO : Test Run:a6e5a151-197a-48d7-86c5-6714ced296b7, Status: UpdateTestRunConfigCompleted.
2017-08-23 16:01:39,598 INFO : Test Run:a6e5a151-197a-48d7-86c5-6714ced296b7, Status: Running.
2017-08-23 16:01:40,267 INFO : Discovering tests...
2017-08-23 16:01:40,268 INFO : Apply test case filter: Name:Category, Operator:Equal, Value:Dothraki.
2017-08-23 16:01:43,370 INFO : Discovered 12 of 337 tests...
2017-08-23 16:01:50,528 INFO : Running: LMS.XUnit.Tests.CreateRelativeCertTest.CreateRelativeCertification.
2017-08-23 16:17:18,856 ERROR: [FAIL] LMS.XUnit.Tests.CreateRelativeCertTest.CreateRelativeCertification: Object reference not set to an instance of an object..
2017-08-23 16:17:18,856 ERROR:    at CSOD.Automation.Selenium.WebControl.get_Enabled() in C:\TeamCity\buildAgent\work\4c2b8c219adc5e13\CSOD.Automation.Selenium\WebControl.cs:line 134
   at LMS.XUnit.ObjectModels.LMSWebControl.get_Enabled()
   at LMS.XUnit.ObjectModels.LMSWebControl.VerifyAndExecute(Action action, Boolean checkEnabled, Boolean checkDisplayed)
   at LMS.XUnit.ObjectModels.LMSWebControl.Click()
   at LMS.XUnit.Tests.CreateCertificationSD.CreateRelativeScheduling()
   at LMS.XUnit.Tests.CreateRelativeCertTest.CreateRelativeCertification()
2017-08-23 16:17:18,869 INFO : Running: LMS.XUnit.Tests.CreateOneTimeCertTest.CreateOneTimeCertification.
2017-08-23 16:24:23,726 INFO : [PASS] LMS.XUnit.Tests.CreateOneTimeCertTest.CreateOneTimeCertification
2017-08-23 16:24:23,735 INFO : Running: LMS.XUnit.Tests.CreateOneTimeCertTest.UserCompletesOneTimeCert.
2017-08-23 16:29:19,516 INFO : [PASS] LMS.XUnit.Tests.CreateOneTimeCertTest.UserCompletesOneTimeCert

Grok filter for logstash config:

filter {


  grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} INFO : Test run configuration: RunId: (?<runid>[^\s]*), Team: (?<Team>[^\s]*), FrameWork:CTFXUnit, Branch: (?<Branch>[^\s]*), Environment:(?<Environment>[^\s]*), Portal:(?<Portal>[^\s]*), Browser:(?<Browser_Type>[^\s]*), EnableDatabaseLog:True, EnableTestRailLog:True, TestRailConfiguration:Milestone: , TestRunType: TestPlan, TestPlan:(?<Test_Plan>[^\s]*), UpdateExistingTestRailRun: True., DebugLogLevel:2, TraitFilters:Name:Category, Operator:Equal, Value:(?<Trait_Value>[^\s]*),"}
	match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} ERROR: \[(?<Test_Run_Status>[^\s]*)\] LMS.XUnit.Tests.(?<Test_Name>[^\s]*)"}
	match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} INFO : \[(?<Test_Run_Status>[^\s]*)\] LMS.XUnit.Tests.(?<Test_Name>[^\s]*)"}
	}

  date {
        match => [ "timestamp", "ISO8601"]
	remove_field => ["timestamp"]
	}

  if "_grokparsefailure" in [tags] 
  
  { drop {} }
	
	
}

Output in Kibana:--

The issue i have is, as can be seen from output in kibana, many of my rows does not have value. One of the row has values which i want to get populated for all the rows depending on condition. Can someone please help me here.

Extreme thanks in advance.

magnusbaeck · August 25, 2017, 5:14am

Please edit your post and format the configuration snippet as preformatted text so it doesn't get mangled.

sunny1 · August 26, 2017, 12:50am

Thanks for quick reply and sorry for not doing it correctly first time itself.

magnusbaeck · August 28, 2017, 7:15am

Only one of the three grok expressions you have are extracting Environment and Trait_Value fields so that could be one reason, but without seeing the full event (expand an event and copy paste from the JSON tab) and the log line that it came from it's impossible to go into further detail.

sunny1 · August 28, 2017, 6:25pm

Magnus, you are right, those fields are extracted only in one of three grok expressions. The reason is as you can see from log file, only one line in the file has those values. Other lines dont have. So my question would be is there way to append the custom tag(trait_value, Environment) which i am creating in one of the grok expression to append to results (match) of second and third grok filter extraction.

Also sorry did not understand what u meant by expand an event and copy paste from JSON tab.

magnusbaeck · August 29, 2017, 5:17am

So my question would be is there way to append the custom tag(trait_value, Environment) which i am creating in one of the grok expression to append to results (match) of second and third grok filter extraction.

I don't believe that's possible.

Also sorry did not understand what u meant by expand an event and copy paste from JSON tab.

In Kibana's Discover view you can click on a triangle to the left of the event to show the whole event. There are two tabs there, one being the JSON tab that shows the raw JSON event. Very useful for debugging.

sunny1 · August 29, 2017, 6:26pm

Thanks Magnus, thought so that it wont be possible so i used alternative with condition in my grok based on source file and then add custom fileds or tags as required. But thanks for taking time out to look into my issue.

system · September 26, 2017, 6:26pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.