# How to filter the logs using logstash

**URL:** <https://discuss.elastic.co/t/how-to-filter-the-logs-using-logstash/136898>\
**Category:** Logstash\
**Created:** [June 21, 2018, 2:41pm UTC](https://discuss.elastic.co/t/how-to-filter-the-logs-using-logstash/136898 "2018-06-21T14:41:27Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nareshreddyp](https://avatars.discourse-cdn.com/v4/letter/n/e56c9b/32.png) [@nareshreddyp](https://discuss.elastic.co/u/nareshreddyp)\
**Post date:** [June 21, 2018, 2:41pm UTC](https://discuss.elastic.co/t/how-to-filter-the-logs-using-logstash/136898/1 "2018-06-21T14:41:28Z")

</div>

I am using filebeat to send the below logs to logstash. But i need to know the best way to filter the fields. Trying csv and grok. But i am not successful to filter properly.

2018-05-21 09:00:03,384 INFO 369.4163.21 swgw400 6 jsp.READY /common/emxMQLNoticeWrapper.jsp clearLimitNotice=true 0  
2018-05-21 09:00:09,527 INFO Checking running requests in 0 pending requests...  
2018-05-21 09:00:09,527 INFO Total requests=65736; Check=3416; Average Pending=0  
2018-05-21 09:00:09,527 INFO ...found 0 running requests in total of 0 pending requests  
2018-05-21 09:00:09,527 DEBUG 0 Running requests  
2018-05-21 09:01:09,527 INFO Checking running requests in 0 pending requests...  
2018-05-21 09:01:09,527 INFO Total requests=65736; Check=3417; Average Pending=0  
2018-05-21 09:01:09,527 INFO ...found 0 running requests in total of 0 pending requests  
2018-05-21 09:01:09,527 DEBUG 0 Running requests  
2018-05-21 09:02:09,528 INFO Checking running requests in 0 pending requests...  
2018-05-21 09:02:09,528 INFO Total requests=65736; Check=3418; Average Pending=0  
2018-05-21 09:02:09,528 INFO ...found 0 running requests in total of 0 pending requests  
2018-05-21 09:02:09,528 DEBUG 0 Running requests  
2018-05-21 09:02:21,451 INFO 369.4164.0 testtest02 99 jsp.READY /emxLogin.jsp ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas 0  
2018-05-21 09:02:21,619 INFO 369.4164.1 testtest02 103 jsp.READY /common/emxSecurityContextSelection.jsp null 0  
2018-05-21 09:02:21,720 INFO 369.4164.2 testtest02 19 jsp.READY /common/emxUIConstantsJavaScriptInclude.jsp null 2199872  
2018-05-21 09:02:21,972 INFO 369.4164.3 testtest02 94 jsp.READY /common/emxSecurityContextSelectionProcess.jsp widgetId=null&SecurityContext=Product+Data+Writer.CompanyFunctionalOrganization.Company+Global 1122512  
2018-05-21 09:02:22,292 INFO 369.4164.4 testtest02 274 jsp.READY /common/emxNavigator.jsp ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas&collabSpace=Company%20Global 0  
2018-05-21 09:02:22,488 INFO 369.4164.6 testtest02 22 jsp.READY /common/emxClientSideInfoProcessing.jsp xhr=0.5139563434887716 1165744  
2018-05-21 09:02:23,579 INFO 369.4164.5 testtest02 1137 jsp.READY /common/emxNavigatorToolbar.jsp toolbar=AEFGlobalToolbar&isPopup=null 51383680  
2018-05-21 09:02:23,694 INFO 369.4164.7 testtest02 3 jsp.READY /integrations/emxIntegrations.jsp null 0  
2018-05-21 09:02:24,179 INFO 369.4164.9 testtest02 90 jsp.READY /common/emxCrossDomainProxy.jsp type=json&cache=-1&1526886144001&headers%5BAccept%5D=application%2Fds-json&headers%5BAccept-Language%5D=sv&headers%5BX-Request%5D=JSON&method=GET&url=https%3A%2F%2Ftest-sit2%2einternal%2eCompany%2ecom%2F3dspace%2Fresources%2FAppsMngt%2Fuser%2Fstartup 2364984  
2018-05-21 09:02:28,947 INFO 369.4164.8 testtest02 5075 jsp.READY /integrations/ief.jsp null 24893984  
2018-05-21 09:02:29,176 INFO 369.4164.10 testtest02 127 jsp.READY /servlet/IEFCommandsServlet isNonIntegUser=true 944760  
2018-05-21 09:02:29,655 INFO 369.4164.13 testtest02 129 jsp.READY /common/emxReadAjaxCall.jsp cmddName=AEFCollabSpace&_=1526886142607 483184  
2018-05-21 09:02:29,849 INFO 369.4164.11 testtest02 587 jsp.READY /common/emxNavigatorContentLoad.jsp ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas&collabSpace=Company+Global 50335304  
2018-05-21 09:02:30,285 INFO 369.4164.12 testtest02 920 jsp.READY /common/emxReadAjaxCall.jsp cmddName=AEFTypesGlobalSearchCommand&_=1526886142606 75429784  
2018-05-21 09:02:59,468 INFO 369.4164.14 testtest02 29481 jsp.READY /common/emxDashboardUser.jsp HelpMarker=emxhelpnewhomepage -1639797624  
2018-05-21 09:02:59,468 INFO 816354264 Memory Used (after gc)  
2018-05-21 09:02:59,468 DEBUG Logging Request com.matrixone.apps.domain.util.XSSInputFilter$FilterRequestWrapper@25c8872c  
2018-05-21 09:02:59,468 DEBUG request.servletPath = /common/emxDashboardUser.jsp  
2018-05-21 09:02:59,468 DEBUG request.pathInfo = null  
2018-05-21 09:02:59,468 DEBUG request.queryString = HelpMarker=emxhelpnewhomepage  
2018-05-21 09:02:59,468 DEBUG request.remoteAddr = 128.87.242.33  
2018-05-21 09:02:59,468 DEBUG request Referrer = [https://test-sit2.internal.Company.com/3dspace/common/emxNavigatorContentLoad.jsp?ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas&collabSpace=Company+Global](https://test-sit2.internal.Company.com/3dspace/common/emxNavigatorContentLoad.jsp?ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas&collabSpace=Company+Global)  
2018-05-21 09:02:59,468 DEBUG request[tomcat.timer.request.id] = 14  
2018-05-21 09:02:59,468 DEBUG request[tomcat.timer.request.memory] = 2456151888  
2018-05-21 09:02:59,468 DEBUG request[tomcat.timer.start] = 1526886149987  
2018-05-21 09:02:59,468 DEBUG request(HelpMarker) = emxhelpnewhomepage  
2018-05-21 09:02:59,468 DEBUG request = text/html, application/xhtml+xml, _/_  
2018-05-21 09:02:59,468 DEBUG request = [https://test-sit2.internal.Company.com/3dspace/common/emxNavigatorContentLoad.jsp?ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas&collabSpace=Company+Global](https://test-sit2.internal.Company.com/3dspace/common/emxNavigatorContentLoad.jsp?ticket=ST-19460-OuODqelTlxdTbwWPoqyc-cas&collabSpace=Company+Global)  
2018-05-21 09:02:59,468 DEBUG request = sv-SE  
2018-05-21 09:02:59,468 DEBUG request = Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko  
2018-05-21 09:02:59,468 DEBUG request = gzip, deflate  
2018-05-21 09:02:59,468 DEBUG request = [test-sit2.internal.Company.com](http://test-sit2.internal.Company.com)  
2018-05-21 09:02:59,468 DEBUG request = testcookie=1; JSESSIONID=E1E85E7C6C30B1AB32F4B116D208AA7A; WT\_FPC=id=69821766-4180-4cf9-bfa6-8dc89a9c8874:lv=1526881671645:ss=1526880951161; has\_js=1; userinfo=%7B%22uid%22%3A%22ERABLO%22%2C%22unit%22%3A%2231550594%22%2C%22role%22%3A%22%22%2C%22location%22%3A%22Sweden%22%2C%22displayname%22%3A%22QW5kZXJzIEJsb20%3D%22%2C%22created%22%3A1526884545%2C%22location\_tid%22%3A%22367%22%2C%22token%22%3A%221527050145%2BS6qsJ\_H1iAVNdNgdwB8OpjtFAGqOLRkMozBHLDxq5mY%22%7D  
2018-05-21 09:02:59,468 DEBUG request = 147.214.118.177  
2018-05-21 09:02:59,468 DEBUG request = close  
2018-05-21 09:02:59,468 DEBUG request = 3dspace  
2018-05-21 09:02:59,468 DEBUG request = [test-sit2.internal.Company.com](http://test-sit2.internal.Company.com)  
2018-05-21 09:02:59,468 DEBUG request = 443  
2018-05-21 09:02:59,468 DEBUG request = https  
2018-05-21 09:02:59,468 DEBUG request = SIT2\_MCS\_Front\_5  
2018-05-21 09:02:59,468 DEBUG thread.name = http-bio-8011-exec-5  
2018-05-21 09:02:59,468 DEBUG memory.delta = -1639797624  
2018-05-21 09:02:59,468 DEBUG session.id = E1E85E7C6C30B1AB32F4B116D208AA7A  
2018-05-21 09:02:59,468 DEBUG session.\_name = 4164  
2018-05-21 09:02:59,468 DEBUG session[tomcat.timer.request.next] = 15  
2018-05-21 09:02:59,468 DEBUG session[tomcat.timer.session.name] = 4164  
2018-05-21 09:02:59,468 DEBUG session[mcadintegration.applet.loaded] = false  
2018-05-21 09:02:59,468 DEBUG session[timeZone] = -1  
2018-05-21 09:02:59,468 DEBUG session[MCADIntegrationSessionDataObject] = com.matrixone.MCADIntegration.server.beans.MCADIntegrationSessionData@b0aaaf3  
2018-05-21 09:02:59,468 DEBUG session[ematrix.mcsurl] = [https://test-sit2.internal.Company.com](https://test-sit2.internal.Company.com)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 21, 2018, 6:11pm UTC](https://discuss.elastic.co/t/how-to-filter-the-logs-using-logstash/136898/2 "2018-06-21T18:11:28Z")

</div>

What configuration(s) have you tried? How would you like to see the log parsed, i.e. what fields do you want to extract?

---

<div class="post-metadata">

**Author:** ![nareshreddyp](https://avatars.discourse-cdn.com/v4/letter/n/e56c9b/32.png) [@nareshreddyp](https://discuss.elastic.co/u/nareshreddyp)\
**Post date:** [June 21, 2018, 8:39pm UTC](https://discuss.elastic.co/t/how-to-filter-the-logs-using-logstash/136898/3 "2018-06-21T20:39:01Z")

</div>

csv {  
skip\_empty\_columns =\> "true"  
separator =\> " "  
columns =\> ["#timeStamp","HTTPMethod","NUMBER","username","timetaken","Request"]  
}

timestamp HTTPMethod Sessionid userid timetaken jsptype url method time  
2018-05-21 09:00:03,384 INFO 369.4163.21 swgw400 6 jsp.READY /common/emxMQLNoticeWrapper.jsp clearLimitNotice=true 0

timestamp METHOD Total Requests=Number checked=number  
2018-05-21 09:00:09,527 INFO Total requests=65736; Check=3416; Average Pending=0

TOmcat request id memory timer start, helper, request from below.

| 2018-05-21 09:02:59,468 | DEBUG | request[tomcat.timer.request.id] = 14 |
| --- | --- | --- |
| 2018-05-21 09:02:59,468 | DEBUG | request[tomcat.timer.request.memory] = 2456151888 |
| 2018-05-21 09:02:59,468 | DEBUG | request[tomcat.timer.start] = 1526886149987 |
| 2018-05-21 09:02:59,468 | DEBUG | request(HelpMarker) = emxhelpnewhomepage |
| 2018-05-21 09:02:59,468 | DEBUG | request = text/html, application/xhtml+xml, / |

How to grok/filter different lines, since the format is different for each line ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 24, 2018, 8:53pm UTC](https://discuss.elastic.co/t/how-to-filter-the-logs-using-logstash/136898/4 "2018-06-24T20:53:34Z")

</div>

You could e.g. use a conditional to check which format the current line has and pick one of two sets of filters.

Or, instead of csv use a single grok filter with two expressions, one for each kind of log message. Logstash will try both expressions (if necessary).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 22, 2018, 8:53pm UTC](https://discuss.elastic.co/t/how-to-filter-the-logs-using-logstash/136898/5 "2018-07-22T20:53:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.