How to filter the logs using logstash

nareshreddyp · June 21, 2018, 2:41pm

I am using filebeat to send the below logs to logstash. But i need to know the best way to filter the fields. Trying csv and grok. But i am not successful to filter properly.

2018-05-21 09:00:03,384 INFO 369.4163.21 swgw400 6 jsp.READY /common/emxMQLNoticeWrapper.jsp clearLimitNotice=true 0
2018-05-21 09:00:09,527 INFO Checking running requests in 0 pending requests...
2018-05-21 09:00:09,527 INFO Total requests=65736; Check=3416; Average Pending=0
2018-05-21 09:00:09,527 INFO ...found 0 running requests in total of 0 pending requests
2018-05-21 09:00:09,527 DEBUG 0 Running requests
2018-05-21 09:01:09,527 INFO Checking running requests in 0 pending requests...
2018-05-21 09:01:09,527 INFO Total requests=65736; Check=3417; Average Pending=0
2018-05-21 09:01:09,527 INFO ...found 0 running requests in total of 0 pending requests
2018-05-21 09:01:09,527 DEBUG 0 Running requests
2018-05-21 09:02:09,528 INFO Checking running requests in 0 pending requests...
2018-05-21 09:02:09,528 INFO Total requests=65736; Check=3418; Average Pending=0
2018-05-21 09:02:09,528 INFO ...found 0 running requests in total of 0 pending requests
2018-05-21 09:02:09,528 DEBUG 0 Running requests
2018-05-21 09:02:21,451 INFO 369.4164.0 testtest02 99 jsp.READY /emxLogin.jsp ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas 0
2018-05-21 09:02:21,619 INFO 369.4164.1 testtest02 103 jsp.READY /common/emxSecurityContextSelection.jsp null 0
2018-05-21 09:02:21,720 INFO 369.4164.2 testtest02 19 jsp.READY /common/emxUIConstantsJavaScriptInclude.jsp null 2199872
2018-05-21 09:02:21,972 INFO 369.4164.3 testtest02 94 jsp.READY /common/emxSecurityContextSelectionProcess.jsp widgetId=null&SecurityContext=Product+Data+Writer.CompanyFunctionalOrganization.Company+Global 1122512
2018-05-21 09:02:22,292 INFO 369.4164.4 testtest02 274 jsp.READY /common/emxNavigator.jsp ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas&collabSpace=Company%20Global 0
2018-05-21 09:02:22,488 INFO 369.4164.6 testtest02 22 jsp.READY /common/emxClientSideInfoProcessing.jsp xhr=0.5139563434887716 1165744
2018-05-21 09:02:23,579 INFO 369.4164.5 testtest02 1137 jsp.READY /common/emxNavigatorToolbar.jsp toolbar=AEFGlobalToolbar&isPopup=null 51383680
2018-05-21 09:02:23,694 INFO 369.4164.7 testtest02 3 jsp.READY /integrations/emxIntegrations.jsp null 0
2018-05-21 09:02:24,179 INFO 369.4164.9 testtest02 90 jsp.READY /common/emxCrossDomainProxy.jsp type=json&cache=-1&1526886144001&headers%5BAccept%5D=application%2Fds-json&headers%5BAccept-Language%5D=sv&headers%5BX-Request%5D=JSON&method=GET&url=https%3A%2F%2Ftest-sit2%2einternal%2eCompany%2ecom%2F3dspace%2Fresources%2FAppsMngt%2Fuser%2Fstartup 2364984
2018-05-21 09:02:28,947 INFO 369.4164.8 testtest02 5075 jsp.READY /integrations/ief.jsp null 24893984
2018-05-21 09:02:29,176 INFO 369.4164.10 testtest02 127 jsp.READY /servlet/IEFCommandsServlet isNonIntegUser=true 944760
2018-05-21 09:02:29,655 INFO 369.4164.13 testtest02 129 jsp.READY /common/emxReadAjaxCall.jsp cmddName=AEFCollabSpace&=1526886142607 483184
2018-05-21 09:02:29,849 INFO 369.4164.11 testtest02 587 jsp.READY /common/emxNavigatorContentLoad.jsp ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas&collabSpace=Company+Global 50335304
2018-05-21 09:02:30,285 INFO 369.4164.12 testtest02 920 jsp.READY /common/emxReadAjaxCall.jsp cmddName=AEFTypesGlobalSearchCommand&=1526886142606 75429784
2018-05-21 09:02:59,468 INFO 369.4164.14 testtest02 29481 jsp.READY /common/emxDashboardUser.jsp HelpMarker=emxhelpnewhomepage -1639797624
2018-05-21 09:02:59,468 INFO 816354264 Memory Used (after gc)
2018-05-21 09:02:59,468 DEBUG Logging Request com.matrixone.apps.domain.util.XSSInputFilter$FilterRequestWrapper@25c8872c
2018-05-21 09:02:59,468 DEBUG request.servletPath = /common/emxDashboardUser.jsp
2018-05-21 09:02:59,468 DEBUG request.pathInfo = null
2018-05-21 09:02:59,468 DEBUG request.queryString = HelpMarker=emxhelpnewhomepage
2018-05-21 09:02:59,468 DEBUG request.remoteAddr = 128.87.242.33
2018-05-21 09:02:59,468 DEBUG request Referrer = https://test-sit2.internal.Company.com/3dspace/common/emxNavigatorContentLoad.jsp?ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas&collabSpace=Company+Global
2018-05-21 09:02:59,468 DEBUG request[tomcat.timer.request.id] = 14
2018-05-21 09:02:59,468 DEBUG request[tomcat.timer.request.memory] = 2456151888
2018-05-21 09:02:59,468 DEBUG request[tomcat.timer.start] = 1526886149987
2018-05-21 09:02:59,468 DEBUG request(HelpMarker) = emxhelpnewhomepage
2018-05-21 09:02:59,468 DEBUG request = text/html, application/xhtml+xml, /
2018-05-21 09:02:59,468 DEBUG request = https://test-sit2.internal.Company.com/3dspace/common/emxNavigatorContentLoad.jsp?ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas&collabSpace=Company+Global
2018-05-21 09:02:59,468 DEBUG request = sv-SE
2018-05-21 09:02:59,468 DEBUG request = Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
2018-05-21 09:02:59,468 DEBUG request = gzip, deflate
2018-05-21 09:02:59,468 DEBUG request = test-sit2.internal.Company.com
2018-05-21 09:02:59,468 DEBUG request = testcookie=1; JSESSIONID=E1E85E7C6C30B1AB32F4B116D208AA7A; WT_FPC=id=69821766-4180-4cf9-bfa6-8dc89a9c8874:lv=1526881671645:ss=1526880951161; has_js=1; userinfo=%7B%22uid%22%3A%22ERABLO%22%2C%22unit%22%3A%2231550594%22%2C%22role%22%3A%22%22%2C%22location%22%3A%22Sweden%22%2C%22displayname%22%3A%22QW5kZXJzIEJsb20%3D%22%2C%22created%22%3A1526884545%2C%22location_tid%22%3A%22367%22%2C%22token%22%3A%221527050145%2BS6qsJ_H1iAVNdNgdwB8OpjtFAGqOLRkMozBHLDxq5mY%22%7D
2018-05-21 09:02:59,468 DEBUG request = 147.214.118.177
2018-05-21 09:02:59,468 DEBUG request = close
2018-05-21 09:02:59,468 DEBUG request = 3dspace
2018-05-21 09:02:59,468 DEBUG request = test-sit2.internal.Company.com
2018-05-21 09:02:59,468 DEBUG request = 443
2018-05-21 09:02:59,468 DEBUG request = https
2018-05-21 09:02:59,468 DEBUG request = SIT2_MCS_Front_5
2018-05-21 09:02:59,468 DEBUG thread.name = http-bio-8011-exec-5
2018-05-21 09:02:59,468 DEBUG memory.delta = -1639797624
2018-05-21 09:02:59,468 DEBUG session.id = E1E85E7C6C30B1AB32F4B116D208AA7A
2018-05-21 09:02:59,468 DEBUG session._name = 4164
2018-05-21 09:02:59,468 DEBUG session[tomcat.timer.request.next] = 15
2018-05-21 09:02:59,468 DEBUG session[tomcat.timer.session.name] = 4164
2018-05-21 09:02:59,468 DEBUG session[mcadintegration.applet.loaded] = false
2018-05-21 09:02:59,468 DEBUG session[timeZone] = -1
2018-05-21 09:02:59,468 DEBUG session[MCADIntegrationSessionDataObject] = com.matrixone.MCADIntegration.server.beans.MCADIntegrationSessionData@b0aaaf3
2018-05-21 09:02:59,468 DEBUG session[ematrix.mcsurl] = https://test-sit2.internal.Company.com

magnusbaeck · June 21, 2018, 6:11pm

What configuration(s) have you tried? How would you like to see the log parsed, i.e. what fields do you want to extract?

nareshreddyp · June 21, 2018, 8:39pm

csv {
skip_empty_columns => "true"
separator => " "
columns => ["#timeStamp","HTTPMethod","NUMBER","username","timetaken","Request"]
}

timestamp HTTPMethod Sessionid userid timetaken jsptype url method time
2018-05-21 09:00:03,384 INFO 369.4163.21 swgw400 6 jsp.READY /common/emxMQLNoticeWrapper.jsp clearLimitNotice=true 0

timestamp METHOD Total Requests=Number checked=number
2018-05-21 09:00:09,527 INFO Total requests=65736; Check=3416; Average Pending=0

TOmcat request id memory timer start, helper, request from below.

2018-05-21 09:02:59,468	DEBUG	request[tomcat.timer.request.id] = 14
2018-05-21 09:02:59,468	DEBUG	request[tomcat.timer.request.memory] = 2456151888
2018-05-21 09:02:59,468	DEBUG	request[tomcat.timer.start] = 1526886149987
2018-05-21 09:02:59,468	DEBUG	request(HelpMarker) = emxhelpnewhomepage
2018-05-21 09:02:59,468	DEBUG	request = text/html, application/xhtml+xml, /

How to grok/filter different lines, since the format is different for each line ?

magnusbaeck · June 24, 2018, 8:53pm

You could e.g. use a conditional to check which format the current line has and pick one of two sets of filters.

Or, instead of csv use a single grok filter with two expressions, one for each kind of log message. Logstash will try both expressions (if necessary).

system · July 22, 2018, 8:53pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.